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ABSTRACT 


This  report  is  primarily  a  review  of  the  state-of-the-art  of 
software  testing  and  verification  with  emphasis  on  techniques  applicable 
to  JOVIAL  J73  programs.  Since  the  project  conterns  a  JOVIAL  J73 
Automated  Verification  System,  the  need  for  such  a  tool,  the  capabil¬ 
ities  for  the  tool,  and  the  high-level  design  of  the  tool  are  also 
described.  Future  capabilities  for  the  tool  are  identified. 
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EVALUATION 

The  purpose  of  this  contractual  effort  was  to  determine  and  specify  the 
required  capabilities  for  an  automated  tt sting  and  verification  system  for 
JOVIAL  J73  software  systems.  The  effort  provided  a  significant  review  of 
the  state-of-the-art  of  software  testing  and  verification,  with  emphasis 
placed  on  techniques  applicable  to  JOVIAL  J73  programs.  The  resulting 
capabilities  were  specified  in  two  separate  documents  -  a  Functional 
Description  and  a  System/Subsystem  Specification,  which  will  be  utilized 
during  the  implementation  phase  of  the  effort.  The  availability  of  an 
automated  testing  and  verification  system  for  JOVIAL  J73  is  significant  in 
that  it  will  enhance  Air  Force  software  development  capability  and  result  in 
a  more  cost-effective  and  reliable  product.  This  effort  was  responsive  to 
the  objective  of  the  RADC  Technology  Plan,  TPO  4G4,  "Higher  Order  Languages.” 
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i  • INTRODUCTION 

Ceneral  Research  Corporation  is  under  a  two-phase  contract  with 
Rome  Air  Development  Center  to  develop  and  implement  an  automated  tool 
to  assist  in  the  testing,  verification,  and  maintenance  of  JOVIAL  J73 
software.  Phase  1  of  this  effort  is  the  study  of  the  state-of-the-art 
of  software  verification  techniques  and  tools  and  the  development  of  a 
functional  description  and  system/ subsystem  specification  for  the  tool. 
Phase  II  of  this  effort  is  the  implementation,  testing,  and  user 
training  period. 


This  report  describes  the  need  for  such  an  Automated  Verification 
System  (AVSj,  results  of  the  state-of-the-art  study,  highlights  of  the 
functional  description  and  system/ subsystem  specification,  and  future 
capabilities  for  consideration.  Additional  reports  resulting  from  this 
effort  are  the  following: 


Phase  I 


Functional  Description 
System/ Subsystem  Specification 
Project  Resource  Document 


Piiase  i  I 


User's  Manual 
Maintenance  Manual 
Test  Plan 

Final  Report:  Implementation  Phase 
Program  Specification 


The  implementation  of  the  A  VS,  called  J/3AVS,  is  expected  to 
conmence  in  May  iVSU.  Figure  1.1  is  a  schedule  of  activities  for  both 
phases  oi  this  oir'ort.  Final  delivery  of  J73AVS  is  scheduled  for 
October  i 981  on  the  I  tel  AS/3  at  Kright-Patterson  AFB  ana  the  DEC  2U  at 
Rome  Air  Development  Center  at  Griff iss  AFB. 


Each  three  months  the  developing  system  will  be  benchmarked;  t iia t 
is,  an  execute  document  (absolute  file)  will  be  created  containing  the 
current  tool-  It  is  expected  that  the  tool  will  have  the  following 
capabilities  at  each  benchmark: 

Benchmark  1  -  Command  ana  control 
Database  management 
Syntax  analysis 

Benchmark  2  -  Structural  analysis 
Instrumental ion 
Static  analysis 
Reaching  set  generation 

Benchmark  3  -  Report  generation 
Path  analysis 
Post-execution  analysis 
Test  history  processing 

The  incremental  benchmarks  are  intendtd  for  our  use  of  J73AVS  to 
analyze  its  own  code,  and  for  limited  use  at  Wright-P.it  terson  by 
Government  personnel  to  give  tlx*  tool  early  exposure- 


ffel 


2  Tilt-  NEED  FOR  J73AVS 

The  need  for  this  automated  verification  system  is  based  upon  the 
emergence  of  a  new  JOVIAL  language  which  will  supersede  t ho  previously- 
approved  JOVIAL  dialects;  the  characteristics  of  the  language  that  make 
it  complex  and  error-prone;  the  type  of  applications  expected  to  be 
written  in  the  language;  and  the  standardization  of  certain  testing 
measures . 

In  an  effort  to  prescribe  a  standard  policy  for  using  computer 
programming  languages  and  for  resting  computer  programming  language 
compilers,  the  Air  Force  issued  AF  Regulation  300-10  in  1  97b.  Two 
JOVIAL  languages,  J3  and  J 73/ 1 ,  were  specified  as  Air  Force  standard 
high-order  programming  languages.  Both  JOVIAL  languages  are  primarily 
designed  for  command  and  control  system  programming.  They  are  es¬ 
pecially  well  suited  to  large  systems  requiring  efficient  processing  of 
a  large  volume  of  data  with  complex  structure. 

Another  JOVIAL  language,  J3B,  evolved  from  J3  for  the  purpose  of 
"'developing  computer  programs  for  the  Boeing  B-l.  Derivatives  of  J3B 
have  been  widely  used  for  avionics  computer  programming.  However, 
JOVIAL  J3B  is  not  a  language  approved  by  AF  Regulation  300-10. 
Therefore,  a  blend  of  J73/1  and  J3B,  plus  additional  features  not  in 
either  language,  has  been  created  to  satisfy  the  programming  needs  of 
both  the  avionics  and  systems  communities.  This  language,  JOVIAL  J/3, 
is  specified  in  MIL-STD-1 589A  and  is  being  refined  for  a  July  1,  1 980 
release.  In  the  spring  of  1980,  AF  Regulation  300-10  is  expected  to  be 
revised  to  cancel  both  J3  and  J 73/ L  languages,  leaving  J/3  as  the  only 
JOVIAL  language. 

It  was  the  desire  to  improve  software  reliability  that  prompted 
the  Air  Force's  request  for  an  Automated  Verification  System  (AVS)  to  be 
developed  and  made  available  as  soon  as  possible  l'ol  lowing  release  of 
validated  JOVIAL  J7J  compileis.  Encouragement  for  an  AVS  and  other 


sup,  ‘i.  tools  also  came  from  the  JOVIAL  Users  Group,  a  body  ol  Inter¬ 
ested  management  and  technical  people  from  industry,  Government,  and  the 
Air  Force. 

2.1  CHARACTER  1ST ICS  OF  J73  PROGRAMS 

As  defined  in  M1L-STD-15U‘JA,  JOVIAL  J73  permits  t he  independent 
processing  of  functional  modules  which  communicate  through  compools  and 
argument  transmission.  J7J  permits  both  recursive  and  reentrant 
procedures  for  effective  multi-processing.  The  language  provides  a  rich 
variety  of  data  types  and  supporting  data  manipulation  functions,  making 
assembly  code  programming  unnecessary  for  most  applications.  However, 
except  for  a  trace  directive  which  supplies  limited  output  facility, 
there  is  no  input/output  capability  in  the  language.  Linkages  to 
assembly  or  alternate-language  routines  are  required  for  input  and 
output . 

Storage  allocation  tor  data  objects  can  bo  both  automatic  (in 
which  storage  is  released  when  control  exits  from  the  program  unit)  or 
static  (in  which  storage  space  is  saved  throughout  the  entire  execution 
of  the  program).  Automatic  allocation  uses  storage  efficiently  but 
makes  certain  datn-usage  errors  possible. 

The  DEFINE  construct  associates  a  name  with  a  text  string  such 
that  whenever  that  name  is  referenced,  the  text  string  replaces  it. 
DEFINE  statements  can  be  nested  and  can  be  redefined  ba.sed  upon  scope. 
Thus,  while  the  capability  is  extremely  useful,  it  adds  another  di¬ 
mension  of  complexity  to  JOVIAL  programs. 

Unfortunately  for  advocates  of  structured  programming,  the  control 
statements  in  JOVIAL  J73  are  not  confined  to  the  "structured  pro¬ 
gramming"  constructs  of  sequential  flow,  IF-THEN-ELSE,  and  WHILE-loops. 
The  language  does  at  least  have  these  constructs,  so  t ha L  piogrammers 
can  write  structured  code  if  they  desire.  However,  unstructured 


statements  as  GOTO,  KALLTHRli,  EXli,  and  AnOKi  .ire  also  permitted.  X no 
GOTO  statement  allows  transfer  from  the  outside  oi  an  iK  or  CASE 


construct  into  the  body  of  the  IK  or  CASE.  GOTO  statements  e.,n  also  ’»« 
directed  to  labels  that  are  external  to  a  program  unit  or  nodule,  n  t  !u- 
iaboL  is  passed  as  a  parameter.  The  KALLTHRU  statement  lilows  control 
to  pass  from  one  CASE  alternative  to  another  wiLhout  making  tiie  test 


normally  required  .it  each  CASE  option.  Tiie  EXI 1  statement  a  lows 


out  of  nn  immediately-enclosing  loop.  The  ABOR'i  staLemeiiL  provides 
transfer  oi  control  to  the  label  specified  in  the  most  recently 
executed,  currently  active  procedure  having  an  ABORT  phrase.  Thus, 


control  transfer  is  not  defined  until  execution  time. 


The  unstructured  control  statements  provide  flexibility  and 
execution-time  efficiency;  but  at  the  same  Lime  they  increase  the  chance 
of  committing  errors  and  make  the  program  more  difficult  to  understand. 
Since  00/.  of  the  total  cost  of  software  is  general  ly  attributed  to 


maintenance,  source  code  scrutability  is  important. 


J7JAVS  wilt  provide  extensive  static  and  data-flow  analysis  to 
detect  and  report  possible  errors  regarding  control  transfers,  data 
contention  due  to  static  allocation,  uninitialized  variables,  struc¬ 
turally  unreachable  code,  potential  infinite  loops,  etc.  Program 
analysis  reports  can  be  generated  on  command  by  the  user  to  describe 
sucli  detailed  information  as  DEFINE  usage,  label  references,  symbol 


properties,  and  global  data, 


2.2  CHARACTERISTICS  OK  APPLICATION  PROGRAMS 


The  programs  that  will  be  implemented  in  JOVIAL  J/3  will  in  of 
similar  nature  to  those  written  in  the  separate  JOVIAL  dialects:  J3, 
J3B,  and  .173/ 1 .  Applications  wiil  be  for  navigation,  information  man¬ 
agement,  flight  controls,  communications,  etc.  The  software  char¬ 
acteristics  of  the  applications  are  varied.  For  example,  flight  control 


software  has  the  following  characteristics: 


-  r~  .**■•*• 


I 


synchronism ion 

distributed  processing 

structurally  simple  control  statements 

simple  data  types 

real-time  processing 

On  the  other  hand,  applications  such  as  command  and  control  systems  have 
very  different  characteristics  such  as: 

-  Dutch  and  interactive  modes 
complex  data  structures 
complex  control  structures 
Large,  monolithic  modules 
non-reai-time  processing 

Avionics  applications  are  often  destined  for  small  on-board 
computers.  For  those  computers  not  having  a  JOVIAL  .»73  or  J73-subset 
compiler,  the  programs  are  developed  on  a  host  machine  and  cross- 
compiled  to  the  target  machine.  As  is  described  in  Appendix  B,  there 
are  no  software  checkout  tools  available  on  these  small  computers,  so  an 
AVS  operating  on  the  host  computer  must  /supply  as  much  assistance  as 
possible  to  detect  errors  in  program  per;  >rmnnce  and  assure  some  level 
of  testing  thoroughness  before  t he  program  is  cross-compiled. 

Command  and  control  systems,  on  the  other  hand,  tend  to  be  very 
Large  (several  hundred  thousand  lines  of  code!.  They  also  tend  to 
evoLve  as  needs  change.  Therefore,  not  only  is  testing  a  major  problem, 
but  nis.o  code  modification  and  retesting  only  what  is  necessary  are 
difficult  tasks.  in  the  face  of  these  problems,  one  of  the  most 
valuable  assets  of  any  software  support  tool  is  the  ability  to  auto¬ 
matically  produce  concise  but  helpful  program  documentation. 
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TEST INC  MEASURES 

The  problem  of  dcterm ining  waen  a  program  is  orror-iree  is  a  tong 
way  from  being  solved.  However,  there  are  tools  available  wiiien  provide 
a  beginning  toward  measuring  the.  tuoroughness  of  testing.  Rather  than 
wait  for  a  solution  to  tile  whole  problem,  Government  ana  industry  should 

be  encouraged  to  take  advantage  of  these  testing  measures  early  in  the 

development  of  software.  The  following  testing  techniques  can  do  used 
as  testing  measures  since  they  provide  quant itai ive  measures  of  vio¬ 

lations  and  other  reported  phenomena  fsuch  as  statement,  branch.,  or  path 
execution  coverage).  Furthermore,  they  are  reliable  in  the  sense  of 
always  producing  the  same  result  (not  relying  on  interpretation): 

1.  Static  analysis  to  detect  cooing  errors  or  illegal  pro¬ 

graming  practices. 


2. 


Assertions  to  specify  legal  or  allowable  performance. 


Statement,  branch,  or  specified  path  coverage  to  measure 
levels  of  execution. 


Software  verification  without  cumpuLer-.iiued  testing  is  extremely 
expensive.  It  would  be  in  the  spirit  of  standardization  to  improve 
reliability  that  the  Air  Force  should  reassess  the  testing  of  computer 
programs,  as  described  in  Air  Force  Regulation  800-14,  to  require  the 
use  of  AVS  tools  in  testing. 
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3  STUDY  OF  AUTOMATED  TOOLS  AND  TECHNIQUES 

This  section  discusses  the  general  problem  of  software  testing, 
describes  existing  methods  and  procedures  for  software  verification, 
provides  a  chart  showing  .  the  main  characteristics  of  currently  oper¬ 
ational  AVS  tools,  and  analyzes  techniques  given  in  the  literature  which 
influenced  the  design  of  the  JOVIAL  J73  tool,  J73AVS. 

3.1  GENERAL  BACKGROUND 

3.1.1  Software  Verification 

Software  system  verification  is  a  critical  problem  recognized  by 
developers,  customers,  and  software  researchers.  The  problem  is 
exceedingly  r  mplex  for  large  systems.  Software  verification  is  a 
process  which  analyzes  requirements,  specifications,  and  implementation. 
In  addition  to  determining  or  proving  consistency  between  each  phase  of 
the  process,  verification  includes  the  problems  of  determining  the 
reliability,  validity,  and  completeness  of  the  testing  phase. 


Since  verification  is  such  a  monumental  problem,  the  approach  to 
improving  the  situation  has  been  to  partition  the  total  process. 
Requirements,  specification,  and  design  language^  have  been  developed  to 
address  the  early  stages  of  software  development,  although  they  have  not 
yet  reached  a  level  of  widespread  acceptance.  Compilers  and  static 
analyzers  attempt  to  verify  semantic  and  other  consistencies  within  the 
implemented  software.  Dynamic  and  symbolic  execution  ana’yzers  address 
software  testing  more  from  a  functional  approach.  Test  data  generation 
assists  with  deriving  complete  test  cases  from  both  structural  and 
functional  viewpoints.  Proof-of-correctness  techniques  attempt  to 
validate  software  in  a  formal  way.  Even  though  the  partitioning 
approacii  has  provided  considerable  progress  in  the  state  of  software 
certification,  each  partition  nevertheless  has  not  achieved  a  high  level 
of  maturity  or  acceptability. 


3. 1.2  Software  Testing 

The  primary  aim  of  testing  is  to  demonstrate  that  a  system  has 
acceptable  performance  in  terras  of  its  specification.  Experience  has 
shown  that  the  software’s  behavior  must  be  considered  over  a  broader 
space  than  the  specified  functions  if  testing  is  to  identify  errors. 

In  Fig.  3.1,  the  universe  of  software  behavior  is  partitioned  in 
two  ways:  the  specified  and  unspecified,  and  the  acceptable  and 

unacceptable.  Experience  with  software  development  tells  us  that  all 
four  of  these  forms  of  behavior  will  exist  when  software  is  declared 
ready  for  testing,  and  all  four  will  continue  to  exist  after  testing  is 
over,  primarily  because  the  testing  process  is  usually  confined  to 
examining  expected  points  in  the  vector  space  of  the  input. 

In  a  typical  software  testing  activity,  the  testing  group  is 
attempting  to  map  these  regions  by  probing  with  single-point  test  cases. 
Their  success  depends  on  the  total  resources  devoted  to  exploring  the 
universe  of  behavior,  and  on  the  effectiveness  with  which  they  apply 
those  resources  in  terms  of  selecting  the  "best"  points  for  testing. 
Effectiveness  can  be  improved  by  the  use  of  a  well-designed  testing 
program  supported  by  automated  tools. 


(  UNSPECIFIED 
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/ 

/  UNACCEPTABLE  J 

Figure  3.1.  Universe  of  Software  Behavior 
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Software  Errors 


Since  the  goal  of  testing  is  the  detection  of  errors,  we  must  know 
something  about  the  characteristics  of  software  errors.  Until  recently, 
there  was  very  little  data  on  the  types  and  causes  of  errors  in  software 
systems.  Recent  studies,  however,  form  a  basis  of  data  from  whicn  we 
can  state  general  characteristics  of  software  errors.*  According  to 
these  studies: 

1.  Most  errors  occur  in  program  logic  or  in  data  access,  not  in 
computation. 

2.  Approximately  half  of  all  errors  are  due  to  errors  in 
specification,  and  the  other  half  are  programming  errors  as 
such. 

3.  Programs  do  not  usually  fail  catastrophically,  but  rather 
errors  degrade  the  program's  performance. 

4.  The  scope  of  errors  is  usually  limited  to  the  one  module 
containing  the  error. 


*  M.  J.  Fries,  Software  Error  Data  Acquisition,  Boeing  Aerospace 
Company  RADC-TR-77-1 30,  Seattle,  Washington,  April  1977.(A039916) 

2 

A.  B.  Endres,  "An  Analysis  of  Errors  and  Their  Causes  in  System 
Programs,"  IEEE  Transactions  on  Software  Engineering,  Vol.  SE-1,  No.  2 
(June  1975)"  ’?•  140-149. 

3 

T.  A.  Thaye  t  al. ,  Software  Reliability  Study,  TRW  Defense  and  Space 
Systems  Groue  76-2266.1.9-5,  Redondo  Beach,  California,  August  1976. 

4 

R.  W.  Motley  and  W.  D.  Brooks,  Statistical  Prediction  of  Programming 
Errors,  IBM  Corp.,  Federal  Systems  Division,  RADC-TR-  77-1 75 , 
Arlington,  Virginia,  May  1977. (A041106) 

^  J.  A.  Dana  and  j.  D.  Blizzard,  Verification  and  Validation  for 
Terminal  Defense  Program  Software:  The  Development  of  a  Software  Error 
Theory  to  Classify  and  Detect  Software  Errors,  Logicon  HR  74012,  San 
Pedro,  California,  May  31,  1974. 
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These  are  only  broad  generalizations  that  one  must  be  careful  in  using; 
there  appear  to  be  many  confounding  factors.  For  example,  the  choice  of 
categories  for  grouping  errors  can  bias  the  results.  Programs  written 
in  high-level  languages  have  different  types  of  errors  than  programs 
written  in  assembly  language.  However,  the  observation  that  a  majority 
of  programming  errors  are  due  to  improper  sequencing  implies  that  a 
large  amount  of  the  testing  effort  should  be  aimed  at  discovering  and 
correcting  these  types  of  errors. 

Sequencing  in  a  program  is  established  by  the  control  statements 
of  the  program  (referred  to  as  the  program's  control  structure).  There¬ 
fore,  it  seems  natural  to  base  the  generation  of  test  cases  and  test 
data  on  techniques  which  analyze  the  program's  control  structure. 
Several  studies  and  tool  developments  have  pursued  this  approach,  with 
most  of  the  efforts  being  applied  to  test  data  generation. 

Functional  Testing 

The  basic  requirement  of  any  system  is  that  it  perform  its 
intended  function.  Functional  testing  is  the  means  by  which  the  actual 
behavior  is  identified;  the  consequences  of  this  behavior  must  be 
related  to  the  intended  function  through  criteria  of  acceptance  derived 
from  the  specification.  (We  ignore  in  this  discussion  the  frequent 
occurrence  that  the  specification  as  interpreted  does  not  represent  the 
intent  of  the  designers).  When  testing  resources  are  limited,  they  are 
applied  to  testing  presumably  representative  instances  of  the  various 
functional  modes  of  the  system.  With  more  testing  resources,  functional 
test  cases  are  usually  expanded  in  an  ad  hoc  manner  in  an  attempt  to 
exercise  more  of  the  alternatives  that  are  recognized  by  the  software. 


Automation  of  functional  testing  usually  takes  the  form  of 
providing  a  means  to  step  through  variations  of  a  basic  test  case. 
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Other  candidates  for  automated  assistance  to  functional  testing 


Analysis  of  special  representations  of  input  space  to  assist 


in  the  selection  of  functional  test  cases 


2.  Static  analysis  tools  that  recognize  assertions  concerning 
functional  behavior  and  check  for  consistency  with  the  code 


Automated  conversion  of  functional  assertions  to  executable 


code  for  execution- time  checking  against  actual  results 


Classification  and  storage  of  input  data,  with  mechanisms 


for  generating  specific  cases 


Classification  and  storage  of  test  results,  with  mechanisms 
for  comparing  test  results  between  cases 


Modification  of  the  input  data  to  map  performance  boundaries 


In  addition  to  the  basic  purpose  of  functional  tests  as  a  means  of 
demonstrating  compliance  with  acceptance  criteria,  these  tests  define 
the  point  of  departure  for  extensions  to  structurally  derived  tests. 


described  in  the  next  section. 


Structure-Based  Testing 


As  Fig.  3. 1  suggested,  it  is  the  nature  of  computer-controlled 
systems  that  they  often  display  modes  of  behavior  that  are  not  ex¬ 
plicitly  identified  in  the  specification.  The  unspecified  behavior  may 
result  from  many  different  causes,  ranging  from  simple  blunders  in 
programming  to  carefully  designed  logic  that  implements  an  erroneous 
interpretation  of  the  specification.  Often  unspecified  behavior  results 
when  the  specification  makes  no  provision  for  a  particular  input 
condition  and  it  is  misinterpreted.  These  unspecified  behaviors  usually 
go  untested  by  functional  testing. 


1 


Structure-based  testing  is  a  means  of  deriving  test  cases  directly 
from  the  software  with  the  intent  of  identifying  program  paths  that  are 
not  tested  by  functional  tests,  and  deriving  test  data  that  will  cause 
those  paths  to  be  executed.  Several  test  tools  now  exist  which  support 
structure-based  testing  by  detecting  which  program  segments  have  been 
executed  by  a  particular  test  case.  The  general  approach  used  with  such 
tools  is  described  below. 

A  graph  model  of  a  program  module  is  developed  which  comprises  an 
input  node,  an  output  node,  and  a  set  of  nodes  which  represent  ait  the 
branch  points  in  the  module.  The  nodes  are  connected  by  links  which 
correspond  to  all  the  straight-line  code  executed  in  the  program  between 
branch  points:  the  "branches,"  "logical  segments"  or  "oecision-to- 

decision  paths.” 

Once  the  graph  model  is  derived,  data  collection  points  are 
automatically  inserted  in  the  links  to  record  which  links  are  exercised 
by  a  particular  test.  Then  the  results  of  a  set  of  tests  are  examined 
to  decide  how  testing  of  unexercised  code  should  proceed.  Most  efforts 
toward  further  automation  of  this  process  have  relied  on  automating  a 
simple  rule  for  test  case  selection  (such  as  finding  a  test  that  reaches 
a  single  unexercised  target  path),  and  then  generating  test  data  for 
that  case.  Several  tools  have  implemented  approaches  to  this  type  of 
automation  (see  Sec.  3.2). 

3.1.3  Graph  Model  Theory* 

This  section  describes  the  foundation  of  graph  model  theory.  This 
foundation  is  used  as  the  basis  for  implementing  data  flow  analysis  (a 
static  testing  procedure),  execution  coverage  analysis  (a  dynamic 
testing  procedure),  and  some  automatic  test  data  generation  techniques. 


1  J.  P.  Benson,  et  ai..  Software  Verification:  A  State-of-the-Art  Re¬ 
port,  General  Research  Corporation  CK-1-638,  March  1975. 


The  use  of  directed  graphs  to  represent  programs  is  a  natural 
outgrowth  of  the  flow  charting  practice.  There  are,  however,  major 
differences  between  a  graph  and  a  flow  chart:  .  When  going  from  a  flow 
chart  to  a  graph  model,  some  information  about  the  program  is  unavoid¬ 
ably  suppressed.  In  a  graph,  attention  is  drawn  to  the  fundamental 
control  structure  of  the  program  (the  “paths”  and  “loops”  in  the 
procedure)  and  not  necessarily  the  calculation  being  performed. 

Program  graphs  are  generally  represented  in  one  of  two  ways.  The 
graph  may  be  described  in  terms  of  basic  blocks,  where  a  basic  block  is 
a  linear  sequence  of  program  instructions  having  one  entry  point  (the 
first  instruction  executed)  and  one  exit  point  (the  last  instruction 
executed).  For  a  JOVIAL  program  S  consisting  of  statements 
S^,S2»«*«,Sn,  a  basic  block  b  is  a  contiguous  subset  of  the  statements 
of  SlSi,si+j,...,Si+jt;k  >  0]  having  the  property  that  no  statement  of  b, 
except  perhaps  S^,  is  the  destination  of  any  transfer-of-control 
statement  anywhere  in  S.  Alternatively,  the  graph  may  be  described  in 
terms  of  branches  (or  decision-to-decision  paths,  DD-paths),  where  a 
branch  is  the  ordered  sequence  of  statements  the  program  performs  as  a 
result  of  the  outcome  of  a  decision  up  until  the  evaluation  of  the 
predicate  in  the  next  decision  statement  encountered.  Figure  3.2 
illustrates  this  definition. 

Depending  on  whether  the  program  graph  is  described  in  terms  of 
basic  blocks  or  branches,  its  nodes  and  edges  have  different  signif¬ 
icance.  When  basic  blocks  are  used,  the  blocks  are  graphed  as  the 

nodes,  and  the  transfers  of  control  as  the  edges,  of  the  graph.  The 

reason  is  as  follows:  basic  blocks  must  be  physically  contiguous 

statements  in  a  program.  They  begin  on  a  branching  [e.g. ,  GOTO  <label>, 
IF(<condition>) j  or  labeled  statement,  and  they  end  on  the  statement 

immediately  preceding  the  next  branching  or  labeled  statement.  Using 
basic-block  terminology,  a  path  through  a  graph  is  described  as  a 

sequence  of  nodes. 
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Figure  3.2.  Diagram  of  a  Branch 
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Alternatively,  a  graph  nay  be  described  in  terms  of  branches,  with 
the  branches  as  the  edges,  and  the  decision  statements  (e.g. ,  IF 
<condition>j  as  the  nodes.  A  branch  nay  include  one  or  more  basic 
blocks  that  are  contiguous  in  terns  of  execution.  For  exaaple,  a  branch 
□ay  include  an  unconditional  GOTO  statement  and  the  sequential  state¬ 
aents  that  follow  its  target  (labeled  stateaent).  Using  branch  tera- 
inology,  a  path  through  a  graph  is  described  as  a  sequence  of  edges. 


The  following  sections  describe  various  techniques  which  identify 
processing  flows  froa  the  graph  oodel  of  the  progran. 

Depth-First  Search 

Depth-first  search  techniques  have  been  applied  to  a  wide  variety 

of  practical  problems  which  can  be  nodeled  as  graphs.  Tarjan1  describes 

algorithns  for  iapleoenting  the  depth-first  search,  and  points  out  that 

the  algorithns  are  linearly  related  to  the  nun be r  of  nodes  and  edges  in 

terns  of  coaputation  tine  and  storage  space.  Depth-first  search 

techniques  can  be  used  vo  identify  a  “spanning  tree"  for  a  graph;  that 

is,  a  subgraph  which  is  a  tree  and  which  contains  ail  the  nodes  of  the 

graph-1  Algorithns  for  traversing  trees  and  visiting  nodes  of  a  tree 

2 

can  then  be  applied  to  the  spanning  tree.  Osterweil  and  Fosdick  have 
inplenented  a  systea  which  performs  data  flow  analysis  using  depth-first 
search  techniques.  By  analyzing  a  systea  of  FORTRAN  nodules  froa  the 
bottoa  of  the  calling  tree  up,  the  systea  classifies  input/output 
variables  at  nodule  interface  boundaries.  Depth-first  search  techniques 
are  applied  to  each  nodule's  prograa  graph  to  determine  the  input/output 
classification  (i.e.,  set  or  used)  for  all  cooaon  variables  and  argu- 
nents  along  all  possible  paths  through  the  oodule.  Several  types  of 
data  usage  errors  can  be  found  while  performing  this  analysis. 


R.  Tarjan,  ”Depth-First  Search  and  Linear  Graph  Algorithns,"  SIAM 
Jour.  Coaputation,  Vol.  I,  No.  2,  June  1972. 

2 

L.  Osterweil  and  L.  D.  Fosdick,  Automated  Input/Output  Variable 
Classification  as  an  Aid  to  Validation  of  FORTRAN  Programs,  Dept.  of 
Conputer  Science,  University  of  Colorado,  Boulder,  Colorado,  CU-CS- 
037-74,  January  1974. 
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Strongly  Connected  Components  • 

Tar j an ^  presents  an  algorithm  using  depth-tirst  search  techniques 

which  identifies  "strongly  connected"  components  of  a  directed  graph,  in 

computational  time  and  storage  space  linearly  related  to  the  number  of 

nodes  and  edges  in  the  graph.  A  strongly  connected  component  of  a 

9 

program  graph  identifies  an  iteration  structure.  Ramamoorthy  describes 
a  procedure  similar  to  this  which  is  to  be  implemented  in  an  Automated 
Evaluation  Validation  System  (AEVS)..  By  conceptually  replacing  strongly 
connected  subgraphs  with  a  subroutine  call  and  a  subroutine  which  con¬ 
tains  the  iteration  structure,  and  then  applying  the  same  procedure  to 
the  program  graphs  of  the  resulting  subroutines,  it  is  possible  to 
abstract  an  internal  calling  tree  from  a  single  program  graph.  Ratna- 
moorthy  suggests  that  this  technique  will  be  especially  useful  for  large 
moduLes  with  complex  iteration  structures.  The  result  of  abstracting 
the  internal  calling  tree  is  that  validation  analysis  can  be  applied  to 
small  non-iterative  subgraphs  (conceptual  subroutines)  of  the  original 
program  graph.  The  problem  of  relating  this  submodule  analysis  back  to 
the  original  module  still  remains  unsolved. 

Schemes 

3 

Sullivan  '  presents  a  different  approach  for  abstracting  a  con¬ 
ceptual  internal  calling  tree  from  the  program  graph  of  a  module.  He 
refers  to  a  program  graph  as  a  scheme.  A  subscheme  is  a  subgraph  of  the 
program  graph  which  has  the  property  that  it  is  a  one-entry/ one-exit 
structure.  An  elementary  subscheme  is  essentially  a  basic  block  or 
DD-path.  The  decomposition  of  a  scheme  by  successive  partitioning  of 


Tarjan,  op.  cit. 

2 

C.  V.  Ramamoorthy,  R.  C.  Cheng,  and  1C.  li.  Kim,  Reliability  and  Integ¬ 
rity  of  Large  Computer  Programs,  University  of  California,  Berkeley, 
ERC-M430,  12  March  1974. 

J.  E.  Sullivan,  Measuring  the  Complexity  of  Computer  Software,  MITRE 
MTR-2646,  25  June  1973. 
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its  proper  subschemes  into  further  subschemes  can  be  carried  out  unt i.. 
aii  subschemes  are  elementary.  The  partitioning  process  creates  a 
conceptual  internal  calling  tree  (in  which  all  possible  submodules  are 
identified).  Sullivan  has  applied  this  representation  of  program 
structure  to  the  problem  of  measuring  the  complexity  of  computer 
software. 


i ntervals 

Compiler  optimization  techniques*  nave  fruitfully  employed  another 

approach  to  graphical  analysis  called  interval  analysis.  Interval 

analysis  is  similar  to  the  techniques  of  identifying  strongly  connected 

subgraphs  and  one-entry/one-exit  subgraphs.  An  interval  is  a  one-entry 

2 

subgraph  which  nay  have  one  or  more  exits.  Hecht  and  Ullman  describe 
an  algorithm  for  identifying  intervals.  A  conceptual  internal  calling 
tree  can  be  abstracted  from  program  graphs  using  this  algorithm. 


W,  Eg 

S  m 


Level-1  Paths 

A  technique  for  identifying  program  flows  explicitly  is  described 
3 

by  Miller.  The  manner  in  which  the  branches  (or  DD-paths)  described 
previously  can  be  combined  in  potentially  legal  ways  in  normal  program 
execution  is  described  by  objects  called  "level-i  paths."  A  level-i 
path  is  a  sequence  of  DD-paths  which  lie  on  the  _ith  iteration  level 
Within  the  program,  i  =  0,  1,  2,....  Because  there  can  be  an  extremely 
large  number  of  distinct  level-i  paths  in  a  program,  it  is  important  to 
consider,  instead,  classes  of  level-i  paths  which  lead  from  the  same 


r.  £.  Alien,  "Control  Flow  Analysis,"  S1GPLAN  Notes,  July  1970. 
v 

M.  S.  iiecht  and  J.  D.  Ullman,  "Flow  Graph  Reducibility SIAM  Jour. 
Computation ,  Voi.  1,  No.  2,  June  1972. 

J  E.  F.  Miller,  Jr.,  A  Hierarchy  of  Program  Testing  Measures,  General 
Resarch  Corporation,  Program  Validation  Project,  February  1974. 


nodes  and  involve  Che  same  kind  and  manner  of  iteration.  Thus,  certain 
forms  of  parallelism  of  DD-paths  along  level-i  paths  are  removed  as  a 
means  to  reduce  the  combinatoric  size  of  ievel-i  path  classes.  The 
result  of  this  reduction  is  to  capture  the  essentially  different,  program 
flows  in  terms  of  a  "principal  level-i  path"  within  each  level-i  path 
class . 

For  example,  Fig.  3. 3  shows  a  set  of  DD-paths  which  corresponds  to 
a  program;  each  DD-path  is  labeled  with  a  letter.  For  thi .  particular 
program  graph,  the  following  level-i  paths  and  path  classes  result: 


1. 


Level-G  path:  ab 


Level-0  path  class:  (cd.e)1!1 

1  i=l 


A. 


Level- 1  path:  fgh 

Level-2  path  class:  {k.)n 

*  l  1=1 


The  level-0  paths  represent  flow  from  tne  input  to  the  output  (from  the 
entry  to  the  exit)  without  iteration;  the  level-i  paths  represent  ith 
level  iteration  "over"  constituent  level-i  paths.  DD-paths  d^  ana 
represent  instances  of  path  parallelism. 


ENTRY 


EXIT 


Figure  3.3.  Sample  Set  of  Level-i  Paths 
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3.1.4  Static  Program  Analysis 

Enhancing  the  diagnostics  reported,  and  providing  information  not 
usually  furnished,  by  a  typical  compiler  leads  to  a  series  of  software 
quality  enhancement  methods  which  can  be  categorized  as  static  analysis. 
These  methods  scan  the  source  text  of  a  program  for  errors  in  syntax  and 
semantics  which  can  be  detected  without  running  the  program  on  a 
computer,  and  provide  consistency  checking  and  documentation  about  the 
definition,  reference,  and  communication  of  data  within  the  program^ 
Some  examples  of  the  supplementary  information  and  error  checking  are: 

Documentation 

Cross  Reference.  A  symbol  cross  reference  for  each  program 

including  symbol  type,  definition,  and  use. 

Local  Storage  Identification.  All  variables  used  as  lecal  storage 
by  a  program  are  identified  by  their  type  and  use. 

Communication  Space  Analysis.  All  variables  which  participate  in 
the  communication  to  other  programs  (parameters,  global  variables) 
are  identified  according  to  their  use  and  type. 

Parameter  Analysis.  Variables  used  as  formal  parameters  to  the 
program  are  identified  and  listed  along  with  their  use  and  type. 

Identification  of  Control  Variables.  Variables  which  affect  the 
flow  of  control  in  a  program  and  where  they  are  referenced  are 
identified. 

Consistency  Checking 

Array  Subscript  Check.  Each  subscripted  variable  reference  is 
checked  against  the  array  declaration. 

Expression  hodc  Check.  A  check  is  made  for  expressions  whose 
arithmetic  mode  changes  when  they  are  assigned  to  a  variable. 

Local  henory  Check.  All  variables  which  have  the  possibility  of 
remaining  defined  over  successive  invocations  of  the  program  are 
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identified  and  their  use  specified  (i.e.., 
variables) . 


JOVIAL  static 


Argument  Check.  Formal  and  actual  parameters  are  checked  for 
inconsistencies  in  type,  mode,  number,  dimensionality,  and  use. 

In  general,  static  analyzers  are  most  useful  in  providing  the  programmer 
with  information  which  wiil  help  debug  programs  more  quickly.  They  do 
this  by  identifying  programming  constructs  which  may  be  legal  but  risky 
and  providing  global,  organized  information  about  the  identifiers  used 
in  this  program. 


Two  basic  types  of  dynamic  program  analysis  are  described  in  this 
section:  analysis  of  statement-level  behavior  and  analysis  of  execution 
coverage.  These  two  techniques  are  well-known,  general-purpose  testing 
aids. 

Statement-level  Analysis 

In  statement-level  dynamic  analysis  ail  program  statements  are 
instrumented  in  order  to  obtain  detailed  information  concerning  the 
program's  internal  behavior.  This  technique  produces  more  detailed  and 
more  source-program-oriented  information  than  such  earlier  techniques 
as  hardware  monitoring,  software  monitoring  ("snapshots"),  and  sim¬ 
ulation  techniques.  Typically,  a  statement-level  preprocessor  auto¬ 
matically  augments  each  source  program  statement  with  other  constructed 
statements  or  invocations  of  run-time  subroutines  which  take  measure¬ 
ments  while  the  program  is  running.  These  measurements  usually  include 
the  values  of  selected  program  variables  and  the  number  and  types  of 
branches  taken.  bxumples  of  the  type  of  data  which  might  be  gathered 
for  a  JOVIAL  J73  program  include: 
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An  execution  count  for  all  statements;  i.e.,  the  number  of 
times  each  statement  was  encountered  during  execution 


2.  For  assignment  statements,  the  initial,  final,  minimum,  and 
maximum  values  of  the  computed  variable 

3.  For  ’F  statements,  a  count  of  the  number  of  times  the 
iF-e>.pression  was  true  and  the  final  value  of  the  1F- 
expression 

4.  Branch  counts  on  each  CASE  statement,  along  with  the  initial 
and  final  values  of  the  case  selector 

5.  The  initial  and  final  values  of  the  loop-control  of  FOR 
sta  tements 

6.  The  number  of  times  a  FOR  loop  was  exited  "normally,"  i.e., 
after  doing  the  specified  maximum  number  of  iterations 

When  the  program  terminates,  summary  reports  are  printed  which 
show  the  ranges  of  the  program's  intermediate  variable  values,  which 
bran-'h*'s  were  taken  ana  with  what  frequency,  and  which  statements  in  the 
program  were  not  executed. 

Execution  Coverage  Analysis 

This  technique  attempts  to  gather  information  on  the  run-time 
sequencing  of  a  program  and  the  flow  of  control  among  the  various 

programs  comprising  a  programming  system.  This  <--.uencing  information 
can  be  represented  at  various  levels  of  detail.  At  the  lowest  level  it 
may  be  a  trace  of  the  statements  executed  by  a  program  when  run  with  a 
particular  testcase,  or  the  sequence  of  branches  executed  by  the 

program.  At  a  higher  level,  the  actual  program  flows  traversed  by  the 
program  may  be  collected  or,  at  a  still  higher  level,  the  dynamic 
calling  sequence  of  procedures  ana  subroutines  in  a  programming  system 
may  be  monitored. 


liie  technique  lor  implementing  program  fl>u  analysis  is  the  same 
as  that  for  statement-level  analysis,  that  is,  soil ware  probes  arc- 
placed  in  the  programs  to  be  monitored  at  the  level  at  which  the 
monitoring  information  is  to  be  gathered.  The  instrumentation  state¬ 
ments  are  simply  invocations  of  run-time  auditing  procedures  which 
record  which  procedure  and  which  control  sequence  or  statement  is  b-*ing 
executed  at  the  time  of  the  monitoring.  A  post -processor  can  then 
reproduce  the  dynamic  flow  of  control  through  a  single  program  or  a 
group  of  programs  at  whatever  level  is  desired.  This  information  is 
useful  in  determining  which  control  flows  and  procedures  were  exercised 
by  which  test  cases  as  a  guide  to  wh.it  testing  remains  to  be  none. 

3*1.6  Automatic  Test  Case  Generation 

Howden*  describes  a  methodology  for  identifying  some  oi  the  test 
cases  for  a  program  automatically.  His  method  first  partitions  the  flow 


of  control  in  a  program  into  standard  classes  ot  paths  much  in  the  same 
2 


way  as  Miller.  Then,  descriptions  of  the  path  classes  by  predicates 
and  relations  are  constructed  in  the  form  of  i  system  of  ineoua L i  t  ies. 

Howden  notes,  however,  that  it  may  not  be  possible  to  derive  those 

descriptions  for  arbitrary  programs  containing  loops.  If  these  des¬ 
criptions  can  be  generated,  the  last  phase  of  the  methodology  is  to 

solve  the  system  of  inequalities  ana  thereby  derive  input  values  which 


will  cause  the  program  to  execute  a  particular  class  of  control  flow. 

1 


The  report  by  Howden  elaborates  on  the  techniques  employed  in  each 
phase  of  the  methodology ,  and  discusses  problems  which  arise  in  im¬ 
plementing  those  techniques.  Phases  one  and  two  have  been  partiailv 
implemented  for  analyzing.  FORTRAN  programs. 


W.  Howden,  Methodology  for  the  Automatic  General  ion  of  Program  Test 
data,  Dept,  ol  information  and  Computer  Science  7K  41,  University  of 
California,  Irvine,  13  February  1974. 
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'*  Mi-liei,  Jr.  and  K.  A.  Melton,  (General  Research  Corporation), 
Automated  Generation  ot  Test  Case  Data  Sets,"  Proceedings  1973  Inter- 
national  Conference  on  Reliable  Software,  Los  Angelos,  21-23  April  1973. 


The  SELECT  system  has  been  Implemented  by  t he  Computer  Science 
Group  at  SKI  to  process  an  experimental  language  which  resembles  a 
subset  of  LISP.  This  system  attempts  to  generate  program  test  cases 
automatically  from  the  program's  semantic  and  control  structure.  In 
contrast  to  Howden's  approach,  SELECT  does  not  initially  identify 
classes  of  program  flow,  but  rather  “executes"  the  program  text  sym¬ 
bolically,  accumulating  information  as  it  goes.  When  a  decision  is 
encountered,  SELECT  keeps  track  of  ail  the  branches  resulting  from  the 
decision  ana  tries  to  remove  chose  branches  which  cannot  be  executed  due 
to  the  outcomes  of  previous  decisions.  In  this  way,  impossible  paths 
are  eliminate!!  as  they  arise.  Two  key  features  of  the  SELECT  system  are 
the  adding  01  “pseudo"  predicates  and  paths  for  array  references  and  the 
ability  co  append  a  Boolean  function  to  the  program  under  test  which 
returns  true  if  the  program  satisfies  its  specification  and  false  if  it 
does  not.  SELECT  ti.-^n  attempts  to  derive  a  test  case  which  makes  this 
function  return  a  false  value,  thereby  giving  an  input  for  which  the 
program  will  fail. 

Semi-automatic  testcase  generation  for  the  purpose  of  extending 

2 

testing  coverage  is  discussed  by  Miller.  in  this  method  it  is  assumed 
that  some  testing  has  been  done  on  the  program  and  the  goal  is  to  derive 
a  test  case  for  executing  a  previously  untested  segment  of  code.  The 
first  step  is  to  identify  a  sequence  of  branches  which  “reach”  the 
untested  code  segment.  This  sequence  is  identified  by  a  flow  analysis 
algorithm  which  operates  on  the  program  graph  model.  The  sequence  of 
branches  corresponds  to  the  sequence  of  statements  which  must  be 


R.  S.  Boyer,  B.  El  spas,  and  K.  M.  Levitt,  “SELECT — A  System  for 
Testing  and  Debugging  Programs  by  Symbolic  Execution,"  Proceedings  1975 
International  Conference  on  Reliable  Software,  Los  Angeles,  California, 


executed  in  order  to  reach  the  untested  code  segment.  This  statement 
sequence  is  then  "backtracked"  (symbolically  executed  in  reverse  order) 
in  order  to  identify  particular  input  conditions  which  will  lead  to  the 
execution  of  the  untested  code  segment. 

3.2  EXISTING  METHODS  AND  PROCEDURES 

There  is  a  wealth  of  published  information  on  software  verif¬ 
ication.  No  one,  we  are  sure,  has  personally  tried  all  the  various 
manual  and  automated  techniques  to  evaluate  them  first  hand.  For  the 
most  part,  software  verification  is  still  a  strictly-manual  process. 
Tools  and  techniques  exist,  but  this  area  of  software  engineering  is  in 
its  infancy.  Most  of  the  tools  and  methodologies  have  severe  restric¬ 
tions  or  require  highly-skilled  persons  to  make  their  application 
successful. 

Some  of  the  current  processes  that  make  up  software  verification 
are  listed  below: 

Requirements 

Requirements  state  what  a  computer  system  should  do  from  the 
user's  viewpoint.  Manual  systems  exist  which  decompose  systems  graph¬ 
ically  (SADT  from  SofTech  and  AXES  from  Higher  Order  Software)  and  which 
tag  requirements  for  later  keying  to  design  and  code  (THREADS  from 
Computer  Sciences  Corporation). 

Specification 

At  least  two  languages  and  tools  exist  for  stating  detailed  spec¬ 
ifications  (Requirements  Specification  Language  -  RSL  -  from  TRW  and 
SPECIAL  from  SRI).  Both  provide  a  rigorous  means  of  stating  spec¬ 
ifications  which  can  be  used  to  detect  inconsistencies.  Both  require 
considerable  expertise  to  use  and  provide  maximum  benefit  when  applied 
to  large  system  developments. 
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HIPO  (Hierarchy  plus  Input -Process-Output)  charts  are  a  manual 
means  of  stating  software  specifications  in  the  context  of  program 
structure. 

Design 

There  are  many  design  methodologies  based  upon  decomposition, 
structure,  data  relationships,  top-down  and  bottom-up  development. 
There  are  also  systems  and  languages  such  as  Process  Design  System 
(PDS  -  from  the  System  Development  Corporation)  and  Process  Design 
Language  (PDL).  PDL  is  a  control-structure  keyword  recognizer. 

Functional  and  Performance  Testing 

Manual  functional  and  performance  testing  is  assisted  by  deriving 
data  from  HIPO  charts,  using  simulations,  obtaining  execution-time 
intermediate-value  printout,  and  running  stress  or  boundary  tests  by 
choosing  data  sets  from  the  specification.  Tool-assisted  functional  and 
performance  testing  can  be  performed  by  using  executable,  logical 
assertions  which  report  inconsistencies  between  specified  and  actual 
behavior;  timing  analysis  where  computer  clock  times  are  reported  at 
module  entries,  exits,  or  branch  points;  or  adaptive  testing  (the 
Adaptive  Tester  from  General  Research  Corporation)  where  performance 
boundaries  are  determined  by  automatically  perturbing  the  input  space. 

Structure-based  Testing 

This  testing  concept  has  been  very  popular  for  providing  a  measure 
for  testing  completeness,  test  data  generation,  error  location,  and 
finding  structural  anomalies.  There  are  a  number  of  automated  tools 
which  perform  branch  testing  (RXVP,  JAVS,  FAVS,  SQLAB,  and  TAP  from  GRC, 
NODAL  from  TRW,  PET  from  McDonnell  Douglas,  Test  Coverage  Analyzer  from 
Boeing)  or  user-specified  sequences  of  statements  (SADAT  from  Kernfor- 
schungszentrum  Karlsruhe  GmbH).  Algorithms  are  being  developed  which 
attempt  to  partition  the  impossible  goal  of  testing  all  control  paths  in 
a  program.  Some  of  these  techniques  are  (1)  identifying  strongly- 
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connected  components  of  a  directed  graph  (Tarjan,  Ramamoorthy) ,  U) 
partitioning  tiie  program  graph  into  subschemes  which  are  s ingio-entry/- 
single-exit  structures  (Sullivan),  (3)  identifying  strongly-connected 
subgraphs  which  are  singie-entry/multiple-exit ,  called  intervals  (Hecht 
and  U liman)  and  (4)  partitioning  the  program  graph  in  terms  of  its 
iteration  level,  called  level-i  paths  (Miller). 


Manual  structure-based  testing  can  be  assisted  by  deriving 
decision  tables  (Coodenough  and  Gerhart)  and  choosing  input  data 
accordingly. 


Structural  anomalies  such  as  deaa  code,  potential  ini  ini te  loops, 
and  infeasible  paths  can  be  determined  by  some  current  AV'S  tools  (ATDC 
from  TRW,  SADAT,  JAVS). 

Consistency  Checking 

The  most  common  techniques  used  to  determine  the  consistency  of 
variables  and  interfaces  are  adding  assertions  to  state  expected  use 
( SQLAB  from  GRC,  ACES  from  UC  Berkeley);  employing  static  analysis 
( AMP1C  from  Logicon,  DAVE  from  University  of  Colorado,  FACES  from  UC 
Berkeley,  RXVP,  FA  VS  and  SQLAB  from  GKC);  using  data  flow  analysis  to 
find  uninitialized  variables  and  interface  inconsistencies  (DAVE,  RXVP, 
SQLAB). 

Test  Data  Generation 

A  great  deal  of  research  energy  has  been  expended  on  developing 
test  data  generators.  So  far,  the  tools  being  developed  to  perform 
automatic  test  data  generation,  such  as  ATTEST  at  the  University  or 
Massachusetts,  are  still  research  orienteu  and  have  had  to  back  off  from 
original  goals.  Other  tools  such  as  test  harnesses  or  the  Adaptive 
Tester  require  input  boundaries  and  invariances  between  variables  to  be 


For  -nan uni  test  data  generation,  Howden  suggests  that  input  data 
be  chosen  to  reflect  special  values  for  the  program.  Ostrand  and 
Wey ulcer  suggest  deriving  data  in  two  phases  based  upon  likely  errors  for 
the  particular  program’s  function  and  likely  errors  for  the  control 
structures  used  in  the  program.  The  possible  worthwhile  approaches  to 
generating  test  data  are  too  numerous  to  elaborate  here. 

Formal  Verification 

Automated  formal  verification  systems  (EFFIGY  from  IBM,  PROGRAM 
VERIFIER  from  USC/IS1,  SID  from  the  University  of  Texas  at  Austin,  SQLAB 
from  GRC,  SELECT  from  SRI)  take  user-supplied  assertions  (called 
verification  conditions)  usually  at  each  branch,  and  symbolically 
execute  them.  The  systems  attempt  to  prove  each  VC  as  it  is  symboli¬ 
cally  executed.  The  process  involves  simplification  of  inequalities 
and,  in  the  case  of  interactive  provers,  the  input  of  occasional  rules 
to  aid  simplification.  Formal  verification  is  still  reserved  for  small 
programs.  Most  of  the  implemented  systems  are  LISP  based. 

Program  Modification 

Tools  which  utilize  a  database  system  and  save  interface  descrip¬ 
tions  or  other  such  system-wide  information  can  be  helpfui  to  support 
program  modification  and  maintenance  activities.  Valuable  information 
for  these  activities  aie  module  interaction  reports,  detection  of  global 
changes,  and  local  updates.  Some  of  the  tools  that  provide  this 
assistance  are  the  Boeing  Support  Software,  SID,  JAVS,  FAVS,  and  SQLAB. 

Documentation 

Automatically-generated  reports  which  provide  information  about 
program  structure,  calling  hierarchy,  local  and  global  symbol  usage,  and 
input  and  output  statement  location  are  very  useful  during  program 
development,  testing,  and  maintenance.  Most  AVS  tools  provide  some  or 
all  of  these  reporting  capabilities. 


J  1 


3.3  CURRENTLY  IMPLEMENTED  TEST  TOOLS 

Fhis  section  presents  a  chart  of  current,  operational  tools  for 
testing,  test  case  generation,  proof  of  correctness,  and  coding  stand¬ 
ards  checking.  There  are  numerous  other  systems  in  various  stages  of 
development,  but  this  chart  is  restricted  to  tools  that  are  of  sub¬ 
stantial  value  and  operate  at  one  or  more  computer  installations. 


_ _ 


4  FUNCTIONAL  DESCRIPTION  OF  J73AVS 

This  section  presents  a  brief  description  of  the  capabilities  of 
J73AVS  and  describes  in  what  phases  of  the  software  life  cycle  the 
capabilities  should  be  used.  A  thorough  description  is  provided  in  the 
Functional  Description. * 

Our  approach  to  the  design  of  an  AVS  for  JOVIAL  J73  is  to  provide 
automated  assistance  for 

program  development 
debugging 
testing 
retesting 

The  approach  excludes 

verification  of  requirements 
verification  of  specifications 
automated  design  aids 

formal  program  verification  (proof  of  correctness) 

The  techniques  for  automating  these  processes  are  not  developed  well 
enough  to  be  reliable  for  general-purpose,  large  software  systems. 

The  specifications  for  the  J73  dialect  and  compilers  include 
rigorous  data-type  checking  and  scope  rules.  The  language  allows, 
however,  constructs  and  control  structures  which  demand  caution  in  their 
usage  (such  as  recursive  and  reentrant  procedures,  jumps  into  certain 
control  structures,  abnormal  exits,  etc.).  Further,  the  language  does 
not  contain  a  mechanism  for  specifying  expected  behavior  or  reporting 
user-specified  abnormalities  (since  there  is  no  input/output  facility). 


1  C.  Gannon  and  N.  B.  Brooks,  JOVIAL  J73  Automated  Verification  System 
Functional  Description,  General  Research  Corporation  CR-1-947,  March 
1980. 
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J73AVS  will  not  duplicate  the  static  consistency  checking  of  the 
compiler,  but,  rather,  provide  the  following  set  ot  facilities  to 
support  program  development,  debugging,  testing,  maintenance,  and 
documentation  of  JOVIAL  J73  programs: 

!•  Logical  assertions  and  timing  probes  (see  ACES,  FAVS,  JAVS, 
RXVPBO,  SQLAB  in  Table  3.1) 

2.  Static  and  data  flow  analysis  (see  ACES,  AMP1C,  DAVE,  FACES, 
FAVS,  PFORT,  RXVPbO,  SQLAB,  STANDARDS  AUDITOR,  SURVAYOR) 

3.  Program  structure  ana  characteristic  reporting  (see  ACES, 
FORTRAN  ANALYZER,  FACES,  FAVS,  JAVS,  NODAL,  PACE,  PET, 
PFORT,  QUALIFIER,  RXVPBO,  SADAT,  SQLAB,  SURVAYOR) 

4.  Statement  performance  dynamic  analysis  (see  FORTRAN 
ANALYZER,  PACE,  PET,  QUALIFIER,  TAP) 

3.  Branch,  path,  and  program  unit  execution  coverage  analysis 
(see  FAVS,  JAVS,  NODAL,  RXVPBO,  SADAT,  SQLAB,  TAP,  TEST 
COVERAGE  ANALYZER) 

6.  Branch  and  program  unit  execution  trace  analysis  (see  JAVS, 
NODAL) 

7.  Execution  timing  analysis  (see  JAVS) 

B.  Structural  retesting  assistance  (see  AMP'O,  ATDC ,  ATTEST, 

DISSECT,  EFFIGY,  FAVS,  JAVS,  RXVPBO,  SQLAB,  SELECT) 


9. 


Test  history  reporting 


J73AVS  will  support  interactive  and  batch  facilities  since  the 
various  stages  of  program  development  through  testing  and  maintenance 
lend  themselves  to  both  modes  of  operation.  The  command  language  will 
be  similar  for  interactive  and  bat  :h  usage,  except  that  the  interactive 
user  will  be  prompted  for  information  where  necessary. 

4.1  SUMMYkY  OF  CAPABILITIES 

A  summary  of  capabilities  is  provided  as  a  flow  diagram  in  Fig. 
4.  1.  This  diagram  describes  the  primary  functions  supported  by  J73AVS 
as  well  as  the  sequence  in  which  they  are  performed.  Figure  4.2  shows 
the  interaction  between  J73AVS  and  the  user.  The  user  can  direct  the 
sequence  of  analysis  activities,  using  information  provided  at  each 

stage  of  processing. 

Although  J73AVS  will  exist  as  a  single  program,  it  is  best 
considered  as  a  collection  of  tools  or  facilities  with  which  the  user 
interacts.  Some  of  the  facilities,  such  as  automated  documentation, 
static  error  reporting,  and  instrumentation,  are  completely  automated 
and  require  only  that  the  user  initiate  the  tasks  by  command.  Other 

processes,  such  as  execution- time  data  collection  or  retesting  assis¬ 
tance,  require  more  information  from  the  user  like  test  data  input  and 
test  target  selection. 

J73AVS  provides  detailed  information  both  statically  and  dynami¬ 
cally  about  the  program  being  analyzed.  It  is  the  role  of  the  user  to 

direct  the  processing  performed  by  J73AVS,  to  analyze  the  output 

produced  by  J73AVS,  and  to  determine  subsequent  action. 

The  role  of  J73AVS  in  the  software  development  cycle  is  to  provide 
automated  assistance  wherever  possible  during  the  program  development 
and  maintenance,  debugging,  testing,  an«  retesting  phases  of  t he  cycle. 
The  user  of  J73AVS  plays  an  active  part  in  the  cycle  as  shown  in  Fig. 
4.3.  This  figure  partitions  tiie  phases  of  the  development  cycle  and 
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r  JOVIAL  J73 
:  SOURCE 


One  or  more  modules  of  JOVIAL  J 73  source  code 
is  input  for  processing  and  analysis.  The 
source  code  may  contain  J73AVS  logical 
assertions  and  timing  probes. 


SOURCE  TEXT 
ANALYSIS, 
STRUCTURAL 
ANALYSIS 


J73  AVS  generates  a  directed  qraph  of  the 
control  structure.  All  syntax,  semantics, 
and  structural  information  is  stored  on 
a  database.  Additional  or  changed  source  code 
causes  an  existing  database  to  be  updated. 


Branch  and 
Path  sequences 
and  test  history 
are  reported. 


STATIC 
ANALYSIS 
OATA  FLOW 
ANALYSIS 


Possible  errors,  warnings,  and  dangerous 
programming  practices  are  reported. 


RETESTING 

ASSISTANCE 


PROGRAM  ANALYSIS 
REPORTING 


Reports  for  program 
documentation,  debugging, 
maintenance,  testing  and 
retesting  are  produced. 


CORRECT 

SOURCE 


STRUCTURAL  X 

ASSERTION 

INSTRUMENTATION 


Software  probes  are  automatically  inserted 
for  dynamic  analysis  of  execution  coverage, 
tracing,  and  performance.  Timing  probes 
and  logical  assertions  are  translated  into 
executable  code. 


TEST  EXECUTION, 
DYNAMIC  DATA 
COLLECTION 


Program  execution  produces  a  data 
collection  trace  file  for  analysis  by  J73  AVS 


ERRORS 

FOUND 


EXECUTION 

ANALYSIS 


Execution  coverage  and  tracing,  statement 
performance,  and  execution  timing  are 
reported  by  testcase  and  by  a  set  of 
testcases. 


TEST  GOALS 
ACHIEVED 


Figure  4.1.  Overview  of  J73AVS 


Figure  4.2.  J73AVS  Interaction  with  User 
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Figure  4.3.  Role  of  J73AVS  in  the  Software  Development  Cycle 
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shows  the  ti«.w  between  the  automated  processing  of  J73AVS  and 
user-supplied  input  or  direction. 

Using  Fig.  4.3  as  a  basis,  a  typical  sequence  of  J73AVS-supported 
processing  can  be  described  as  follows: 

1.  JOVIAL  J73  source  text  is  generated  and  provided  to  J73AVS 
as  one  or  more  compilable  modules. 

2.  J73AVS  produces  program  analysis  reports  showing  control 

structure,  symbol  usage,  calling  hierarchy,  etc.,  as  well  as 
a  static  analysis  report  showing  errors  and  dangerous 
programming  practices. 

3.  Using  the  reports  as  a  guide,  the  source  modules  can  be 

modified  or  new  modules  added  to  the  program. 

4.  J73AVS  identifies  the  interaction  of  the  new  or  modified 
modules  with  the  rest  of  the  program;  this  information,  in 
turn,  is  used  as  the  basis  for  modifying  other  modules. 

5.  For  dynamic  debugging,  the  program  is  instrumented  by  J73AVS 
and  executed  with  an  initial  test  case  supplied  by  the  user. 

6.  J73AVS  reports  assertion  violations,  if  any,  and  generates 
an  evaluation  of  statement  and  variable  performance. 

7.  Using  this  evaluation,  the  user  may  choose  to  generate 

additional  test  data  to  pinpoint  errors  or  instrument  other 
modules  for  additional  dynamic  debugging. 

8.  The  same  procedures  of  test  data  generation,  instrumen¬ 

tation,  and  execution  are  performed  for  testing  but  for  a 
different  goal:  rather  than  detecting  and  locating  errors, 
testing  aims  to  demonstrate  the  absence  of  errors.  There¬ 
fore,  J73AVS  produces  execution  analysis  reports  in  terms  of 
the  thoroughness  of  execution  coverage. 
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9.  The  user  evaluates  execution  coverage  and  other  program 
performance  output,  along  with  the  program's  own  execution 
results  and  the  program  specification,  to  determine  if 
testing  is  complete. 

10.  J73AVS  provides  branch  sequence  information  to  retest 

targets  chosen  by  the  user.  A  te  ,t  history  of  execution 
coverage  and  assertion  violations  assists  the  user  in 
choosing  targets  for  retesting. 

Program  Development  and  Maintenance 

Executable  assertions  permit  a  programmer  to  specify  expected 
behavior.  J73AVS  supports  the  technique  of  embedding  programmer- 
specified  assertions  into  the  code  through  the  use  of  the  ASSERT  keyword 
followed  by  any  legal  logical  (Boolean)  expression.  Logical  assertions 
can  be  used  for  execution-time  exception  reporting,  stress  testing,  test 
data  generation  filtering,  and  (left  as  comments  in  the  source  code) 
stating  in-line  specifications. 

To  assist  with  reliable  system  development,  maintenance,  and 
documentation,  J73AVS  will  provide  substantial  program  analysis  re¬ 
porting  on  structural  hierarchy,  symbol  usage,  invocations,  certain  J73 
constructs,  and  system  characteristics.  The  user  has  control  over 
obtaining  high-  or  low-level  information  through  the  command  language. 
The  types  of  program  analysis  reporting  include  the  following: 

indented  source  listing  with  control  structure  identi- 
f  icat ion 

symbol  cross  reference  with  set-use  information 

conpool  symbol  description 

properties  of  all  or  specified  symbols 

declaration  and  reference  of  labels  (statement  names) 

declaration  and  reference  of  user-defined  data  types 
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declaration  and  reference  of  constants 


usage  of  external  reference  (REF)  and  definition  (DEF) 
declaration  and  reference  of  DEFINE  text  strings 
description  of  program  units  on  the  database 


Debugging 

Normal  compilation  using  JOVIAL  J73  compilers  will  detect  many 
syntax  and  semantic  errors.  Additional  errors  such  as  uninitialized 
variables,  possible  infinite  loops,  unreachable  code,  certain  improper 
constructs,  and  dangerous  coding  practices  (like  transferring  into  CASE 
or  IF  statements)  will  be  reported  by  J73AVS.  The  user  can  command 
different  levels  of  static  reporting. 

Dynamic  debugging  will  be  supported  by  statement  execution 
performance  and  assertion  exception  reporting.  Statement  execution 
performance  provides  execution  counts  of  statements,  values  and  ranges 
of  variables  in  assignments  and  loops,  and  the  execution  behavior  of  IF 
statements.  This  debugging  information  appears  adjacent  to  the  source 
statements  themselves,  which  assists  the  task  of  code  correction. 
The  execution  of  timing  probes  (inserted  by  command)  can  be  reported  in 
the  debugging  performance  report  at  the  user's  request. 

When  the  program's  execution  behavior  deviates  from  the  acceptable 
logical  behavior  specified  by  the  embedded  assertions,  it  will  be 
reported  during  execution.  The  user-supplied  assertions  remain  rela¬ 
tively  transparent  to  the  program  until  they  are  violated;  at  that  time 
the  violation  is  reported  along  with  the  source  statement  number  where 
the  violation  occurred. 


Testim 


When  used  in  conjunction  with  static  checking  and  statement-level 
performance  analysis,  structure-based  testing  can  uncover  errors  due  to 
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untested  branches  (where  a  branch  is  a  control  flow  outcome  due  to  a 
decision  statement)  or  improper  sequences  of  branches.  J73AVS  will 
provide  execution  tracing  of  program  units  and  branches  and  execution 
coverage  analysis  of  program  units,  branches,  and  sequences  of  branches 
(paths).  Further,  J73AVS  will  assemble  the  timing  information  from 
program  unit  tracing  and  user-supplied  timing  probes  into  an  execution 


Liming  report. 


Although  an  AVS  can  provide  an  objective  measure  of  testing 
thoroughness  in  terms  of  statement  or  branch  execution  coverage, 
frequently  errors  in  software  are  overlooked  during  testing  because  only 
certain  sequences  of  branches  are  ever  executed.  Obviously,  it  is 
generally  impossible  to  define  all  paths  in  programs  because  of  loops. 
Furthermore,  the  most  likely  subset  of  paths  to  test  can  best  be 
identified  by  a  person  familiar  with  the  function  of  the  program.  The 
most  efficient  role  of  an  AVS  in  this  regard  is  to  identify  the  set  of 
control  paths  between  two  statements  in  a  program  unit  (an  invokable 
unit  of  code)  to  which  the  human  tester  attaches  importance.  Of  the  set 
of  paths  identified  by  the  tool,  the  user  can  choose  those  that  are  to 
be  analyzed  for  coverage  during  execution.  If  the  set  of  paths  is  too 
large  to  enumerate,  a  descriptive  message  will  be  issued  and  the  user 
allowe_  choose  another  pair  of  statements  for  path  identification. 


Retesting  Assistance 


Retesting  software  is  performed  when  analysis  shows  that  prior 
testing  is  inadequate  (insufficient  branch  coverage,  not  all  functions 
demonstrated,  etc.)  or  when  program  changes  have  taken  place.  The 
proper  approach  to  take  in  retesting  is  highly  dependent  upon  the 
characteristics  of  the  program  being  tested  as  well  as  the  measures 
being  used  to  evaluate  testing  completeness.  A  detailed  methodology  for 
testing  and  retesting  software  for  the  purpose  of  improving  structural- 
testing  completeness  will  be  given  5n  the  User's  Manual. 


In  order  co  determine  the  sequences  of  branches  wiiicn  must  be 
executed  in  order  to  reach  an  untested  branch  or  statement,  the  user  can 
request  that  the  "reaching  set"  be  computed  between  two  specified 
statements  (or  from  the  program  unit's  entry).  The  u^<-  can  also 
request  a  list,  in  terms  of  branches,  of  all  control  paths  between  two 
specified  statements.  If  certain  loop  structures  make  this  list  impos¬ 
sible,  subsets  of  the  paths  will  be  identified. 

With  the  control  flows  identified,  the  user  can  backtrack  through 
the  program  to  the  input  space,  using  statement  execution  performance 
reports,  module  interaction  and  invocation  reports,  and  execution 
coverage  information  for  each  testcase  to  assist  in  developing  new  test 
data.  Unfortunately,  automatic  test  data  generators  which  use  symbolic 
execution  are  not  yet  developed  to  the  point  of  being  general-purpose, 
easy  to  use,  or  reliable. 

The  cumulative  test  coverage  history  maintained  by  J73A7S  will  be 
useful  in  attaining  testing  goals  and  determining  targets  for  retesting. 
Program  unit  and  branch  coverage  information  will  be  saved  in  a  concise 
way  on  the  database  for  each  test  case.  The  results  of  subsequent 
execution  runs  can  be  added,  providing  a  cumulative  report  of  all  tests. 
Also  saved  in  the  history  database  table  will  be  any  assertion  vio¬ 
lations  that  occur.  This  will  provide  a  mechanism  for  identifying  which 
input  test  case  caused  a  violation. 

Unfortunately  there  is  no  technique  that  can,  in  general,  echo 
back  to  the  user  what  the  input  for  each  testcase  is.  Paragraph  4.1.1. 3 
of  the  Statement  of  Work  (PR  No.  B-9-3278)  requested  the  identification 
of  input  test  data  used  for  each  testcase,  but  this  can  be  done  only  in 
trivial  cases  such  as  input  on  a  single  file.  In  complex  programs,  data 
are  input  from  a  variety  of  external  sources  such  as  databases,  sub¬ 
routine  parameters,  and  files.  J73AVS  distinguishes  separate  testcases 
(as  defined  by  the  user)  in  its  post-execution  analysis  reports  but  does 
not  print  test  data  input  used  to  drive  each  testcase. 


4.2  J 73A VS  OPERATION 

J/3AVS  will  be  implemented  to  operate  in  both  batch  and  inter¬ 
active  modes.  This  versatility  provides  the  user  with  the  ability  to 
customize  a  debugging  and  testing  strategy  to  his  own  software. 
Depending  upon  the  test  object  (program  being  tested}  and  testing  goals, 
the  sequence  or  J?JAVS  operations  may  be  varied.  Figure  4.1  showed  a 
typical  flow  of  operations,  beginning  with  analysis  of  previously 
unanalyzed  code  and  pro<_ec-ding  until  some  testing  goal  is  realized. 

The  functions  of  J73AVS  will  be  driven  by  user  command.  The 
command  syntax  wiii  be  similar  for  both  batch  and  interactive  codes  of 
operation.  The  command  language  is  made  up  of  specification  and 
operation  commands.  Specification  commands  consist  of: 

BATCH 

to  notify  J73AVS  of  the  mode  of  operation  (the  default  -  no  command  -  is 
interactive  node)  and  the  text  specif ication  commands: 

MODULES  =  name,....  FOR  MODULES  =  name,... 

(Two  or  more  commands) 

END  FOR 

UN 1 75= name,....  FOR  UNITS  =  name,... 

(Two  or  more  commands) 

END  FOR 

SYSTEM  FOR  SYSTEM 

(Two  or  more  commands) 

END  FOR 


J73AVS  operation  commands  control  six  major  functional  capabilities: 
read  source  text  and  buiid  database,  perform  static  and  data  flow 
analysis,  prepare  program  analysis  reports,  instrument  the  source  text 
for  dynamic  analysis,  perform  post-execution  analysis,  and  provide 
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reeesting  assistance.  These  comm*.  ds  will  have  the  following  syntax 
(defaults  are  underlined): 


i 

i 

1.  Read  JOVIAL  J/3  Source  code  - 
Command:  i\cAD{,  CHANGES) 

2.  Static  and  data  flow  analysis  - 

Command:  STATIC  {.  LOCAL /GLOBAL, OFF  = (ERRORS, 

WARNINGS , MESSAGES , SYMBOLS ) , SUMMARY /FULL } 

3.  Program  analysis  reporting  - 
Commands:  LIST 

CROSS  REF  ( .MATRIX, SETUSE, NAMES  =name,...} 

INVOCATIONS  (.MATRIX, TREE, BANDS .SOURCE) 

•  COM POOL  {.XREF, SOURCE) 

SYMBOLS  {,  LIST  .PROPERTIES,  SOURCE,  NAMES  =na  me, ...  ) 
f  LABELo  {, LIST, XREF, SOURCE) 

TYPE  (.LIST, SOURCE) 

CONSTANTS { , LIST, SOURCE) 

REFDEF  (.LIST, SOURCE) 

DEFINE  {.LIST, SOURCE) 

DATABASE  {.UNITS, DESCRIPTION) 

4.  Instrumentation  for  dynamic  analysis  - 
Commands  =  INSTRUMENT  {  ASSERTIONS .STATEMENTS , 

COVERAGE  =  BRANCH /ENTRY, TRACE =BRANCH/ENTRY } 

TRACESET  (UNIT=name .LOCAL /SUBORDINATES, ) start  smt,  stop  smt)) 
NEWTEST, UNIT  =  name, smt. 

ENDTEST ,  UNIT  =  name,  smt. 

!  STARTCLOCK,  UNIT  =  name,  smt. 

STOPCLOCK,  UNIT  =  name,  smt. 


Post-execution  analysis  - 

Commands  =  COVERAGE {  ,  LNTRY  .BRANCti ,  SI AT EM ENT ,  NOTH IT . 


n ITS =BRANCh /PATHS (path  no., path  no.,...)} 

TRACE t , ENTRY /BRANCH  } 

PERFORMANCE  {, ASSERTIONS) 

TIMING 

fc.  Retesting  assistance  - 

Commands  =  SET PATH , UNIT-name .bRANCHES-braneh  1 , branch!, 
{.branch!,...}*  {,RESE1} 

PATHS , UNIT =unme , START =smt . , STOP=smt • ,  LI MlT=number** 
BRANCHES. UNlT=name , START=smt . , STOP=smt .  { , ITERATIVE} 
HISTORY  {.RESET} 

Ther«.-  are  two  additional  commanas:  HELP  and  SAVE.  The  HELP 
conmnnd  is  for  t he  interactive  user  to  provide  command  syntax  assis¬ 
tance.  The  SAVE  command  is  used  to  save  the  current  contents  ot  the 
database.  The  function  at  each  command  is  briefly  described  in  lable 
4-1.  A  thorough  description  ot  each  command,  along  with  sample  usage 
and  output,  is  provided  in  the  Functional  Description. 

Figures  4.4  through  4.1k  snow  input-process-output  for  the  major 
functional  capabilities.  Figures  4.4  aid  4o  illustrate  the  flow  of 
information  tor  commands  READ  and  STATIC.  Figures  4.6  through  4.9 
illustrate  instrumentation  and  execution  of  instrumented  modules. 
Figures  4.10  and  4.11  illustrate  pos.  -execution  analysis,  and  Fig.  4.12 
shows  program  analysis  reporting. 


*  Repetition  of  branch  sequences  is  denoted  by  enclosing  the  branch 
numbers  in  parentheses. 

*x  Tiie  default  number  ot  paths  is  50. 

Note:  All  commands  can  be  abbreviated  to  the  first  four  letters  of 

each  keyworu. 


TABLE  4.1 

J73AVS  COMMAND  LANGUAGE 


Comm  anil 


Parameters 


•STATIC 


CHANGES 


LOCAL /GLOBAL, 

OFF* (ERRORS, WARNINGS, 
MESSAGES, SYMBOLS) , 
FULL /SUMMARY 


CROSS  REF  MATRIX, SKTUSE, NAMES = 

niitnu  y  •  •  *  ♦ 

IN  VOCAT  LONS  NA  1'R  1 X ,  TREK  ,  BANDS  .SOURCE 


COM POOL  XREF, SOURCE 


SYMBOLS 


LABELS 


LIST, PROPERTIES, SOURCE , 
NAMES* name , . . . 

LIST, XREF, SOURCE 


LIST, SOURCE 


CONSTANTS  LLaT, SOURCE 


REFDEF 


DEFINE 


LIST, SOURCE 


LI  ST, SOURCE 


DATABASE  UNITS,  DESCR  I P  I’ ION 


Fund i on 


Read  IOViAL  J7.S  source.  Bnii< 
database.  luentiiy  r Santee, 
modules  on  in iabx.se . 

Perform  .static  and  data  I  low 
ana l vs  is . 


Proonce  indent on  source 
list  i  ng . 

Produce  svi’ibol  cross  referoniv 


Pr ounce  reports  describing 
program  unit  invocation 
si  rue  lure. 

Produce  reports  descr .  aing 
con pool  symbol  usage. 

Produce  reports  describing 
symbol  attributes. 

Produce  reports  describing 
statement  n  lines  (labels). 

Produce  reports  describing 
user-dot inod  datatypes. 

Produce  reports  describing 
constant  datatypes. 

Produce  reports  descrioing 
instances  of  REF  and  DEF 
spec i f icat ion. 

Produce  reports  describing 
instances  or  DEFINE  dec  tar. it  ion 
and  reference. 

Product  reports  describing  the 
program  units  stored  m  the 
current  database. 
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Command 


TABLE  A. 1  Continued 
Parameters 


Function 


INSTRUMENT  ASSERTIONS, STATEMENTS , 

COVERAGE=BRAHCH/ ENT R Y , 
TRACE3 BRANCH/ ENTRY 


TRACESET  U.\TT=name , 

LOCAL/ SU  BOR  0  L  NATES , 
(start  sint,  stop  smt) 

NEW TEST  UN lT=name , smt . 

HNUTEST  UN IT=aame , smt . 


STARYCLOOK  UNI T= name , smt . 


STOPCLOCK  UNIT3 name , smt . 


COVERAGE  ENTRY , BRANCH , ST A  f EMENT , 

NOTH i C , 

ri  £TS= BRANCH / P  ATHS  ( pa  th  no , , 
TRACE  ENTRY /BRANCH 


PERFORMANCE  ASSERT  LONS 


TIMING 


SETPATH 


PATHS 


UN lT=name , BRANCHES3. . . , 
RESET 


UN  lT=:iane ,  START3 smt . , 
STOP=smt.,  LlMIT=no. 


Inserts  software  probes  into 
the  source  code  to  collect  data 
during  execution.  Translate 
assertions  into  executable 
code. 

Instrument  each  branch  between 
the  specified  statements , in¬ 
cluding  branches  in  subordinate 
program  units. 

insert  a  testcase  boundary  at 
the  specified  statement. 

Insert  an  end-of-all-testcases 
probe  at  the  specified 
statement. 

Insert  a  "start"  system  clock 
probe  at  the  specified 
statement. 

Insert  a  "stop"  system  clock 
probe  at  the  specified 
statement. 

Produce  post-execution  analysis 
reports  describing  statement, 

)  branch,  or  path  coverage. 

Produce  a  post-execution 
tracing  report  for  branches  or 
program  unit  entries  and 
returns . 

Produce  a  post-execution 
statement  performance  report, 
including  assertion  violations. 

Produce  an  execution  timing 
analysis  report. 

Store  the  specified  branch 
sequences  in  the  database  as 
paths. 

Identify  the  paths  between  the 
two  specified  statements. 


I 
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TABLE  4. 1  Continued 


Command 

Parameters 

Function 

BRANCHES 

UNIT=name , START=smt . , 
STOP=smt. ,  ITERATIVE 

Generate  a  reaching  set  of 
branches  between  the  two 
specified  statements. 

HISTORY 

RESET 

Produce  a  execution  coverage 
report  for  all  testcases. 

Reset  the  database  coverage 
history  table. 

HELP 

Assist  with  command  syntax. 

SAVE 

Save  the  current  database. 

BATCH 

Indicate  batch  mode  of 
operation. 

MODULES 

name  , « •  •  • 

Specify  one  or  more  modules  for 
the  following  command 
processing. 

FOR  MODULES 

name  t •  •  • 

Specify  one  or  more  modules  for 
the  following  set  of  commands. 

UNITS 

name  }  • 

Specify  one  or  more  program 
units  for  the  following 
command. 

FOR  UNITS 

name  ,  • .  • 

Specify  one  or  more  program 
units  for  the  following  set  of 
commands. 

SYSTEM 

Specify  all  program  units  in 
the  database  for  the  following 
command. 

FOR  SYSTEM 

Specify  all  program  units  in 
the  database  for  the  following 
set  of  commands. 

END  FOR 

Conclude  the  set  of  commands. 

END 

Conclude  J73AVS  processing. 

I 
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NEWTEST  STARTCLOCK 
ENDTEST  STOPCLOCK 


Figure  4.6.  Structural  Instrumentation 
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Figure  4.7.  Assertion  Instrumentation 


INSTRUMENT, 

STATEMENTS 


Figure  4.8.  Statement  Performance  Instrumentation 
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Figure  4.9.  Test  Execution  Processing 


Figure  4.10.  Structural  Testing  Analysis 
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Figure  4.x2.  Program  a.ialysis  Reporting 


p-vg«*af 


DESIGN  OF  J73AVS 


J73AVS  will  be  made  up  of  a  Nucleus  and  set  of  independent 
function  processor  segments.  Each  of  the  segments  can  correspond  to  an 
overlay  segment.  The  Nucleus  can  make  up  the  core-resident  root  (or  the 
first  level)  of  the  overlaid  program,  although  to  minimize  storage 
requirements,  some  Nucleus  routines  will  be  loaded  in  secondary  over¬ 
lays.  Each  of  the  other  functions  makes  up  a  second-level  segment.  The 
following  is  a  brief  description  of  each  functional  segment: 


Command  Decoding  and  Control:  Process  user  input  commands,  output 
interactive  response,  and  successively  return  each  command  to  the 


overlay  controller. 


Initialization  nd  Wrapup:  Upon  run  initialization,  open  files, 
initiate  execution  of  the  storage  manager,  and  set  various  global 
data;  upon  run  termination,  close  files  and  (for  batch  mode) 


produce  report  index. 


JOVIAL  J73  Source  Text  Analysis:  Read  JOVIAL  J73  source  and 
perform  lexical  scan,  token  recognition,  symbol  classification, 
and  structural  pointer  construction. 


Structural  Analysis:  Build  program  graph,  store  branches,  and 

compute  single-entry/ single-exit  reduction  history  used  in  data 


flow  analysis. 


Supplementary  Table  Building:  Build  tables  needed  for  module 

dependence  reporting  and  cross  references. 


Program  Analysis  Reporting:  Produce  selected  reports  at  user 


command. 


Instrumentation:  Insert  probes  at  program  unit  entries,  exits, 

branches,  and  statements  (depending  upon  type  of  instrumentation 
selected);  define  new  testcase  or  end  of  all  testcases;  expand 


assertions  into  executable  code. 
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Structural  Testing  Analysis:  Analyze  run-time  execution  trace 
file,  produce  coverage  and  trace  reports,  and  update  test  history 
table. 


Statement  Performance  Analysis:  Analyze  run-time  trace  and 
instrumentation  statement  descriptions  and  produce  statement 
performai.  >  reports. 

Execution  Titnin.-  Analysis:  Analyze  run-time  execution  trace  and 
produce  timing  repi.  "t. 

Path  Generation:  Determine  the  set  of  paths  between  specified 

statements  and  store  paths  into  database. 

Branch  Reaching  Sets:  Generate  sets  of  branches  that  reach  a 

specified  statement. 

Test  History:  Generate  a  test  coverage  history  report  or  reset 
the  history  table. 

Print  Services:  Print  the  contents  of  specified  database  tables. 

Table  5.1  lists  the  functional  processor  segments  along  with  the 
associated  user  commands  which  invoke  each  segment.  The  Nucleus 
consists  primarily  of  database  management  facilities.  The  segments 
loaded  at  a  particular  time  during  a  run  will  depend  upon  the  type  of 
processing  requested  by  the  user  through  commands. 
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TABLE  5.1.  J73AVS  FUNCTIONAL  PROCESSOR  SEGMENTS 


Command  Keyword 

Segment  No. 

Functional  Processor  Segment 

All  commands 

1 

Coaaand  Decoding  and  Control 

All  commands 

2 

Initialization  and  Urapup 

READ 

3 

JOVIAL  J73  Source  Text  Analysis 

READ 

4 

Structural  Analysis 

STATIC 

5 

Static  and  Data  Flow  Analysis 

INVOCATIONS , CROSSREF  6 

Supplementary  Table  Building 

Reports* 

7 

Program  Analysis  Reporting 

NEWTEST 

ENDTEST 

STARTCLOCK 

>  d 

Instrumentation 

STOPCLOCK 

'  INSTRUMENT** 

TRACESET 

COVERS  ! 

f 

! 

TRACE 

9 

Structural  Testing  Analysis 

PERFORMANCE 

10 

Statement  Performance  Analysis 

TIMING 

11 

Execution  Timing  Analysis 

SETPATHS, PATHS 

12 

Path  Generation 

BRANCHES 

13 

Branch  Reaching  Sets 

HISTORY 

14 

Test  History 

LIST, PRINT*** 

i5 

Print  Services 

* Cocraands  CROSSREF , INVOCATIONS , COMPOOL , SYMBOLS , LABLES , TYPE , CONSTANTS , 
REFDEF , DEF 1XE , DATABASE 

**Structural  inst runentat ion  (paraaeters  COVERAGE  and  TRACE)  and 
statement  performance  instrumentation  (parameter  STATEMENTS)  can  be 
sub-overlays. 

***Database  table  print  package,  primarily  for  J73AVS  development  and 
maintenance  and  for  source  listing  reports. 
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The  design  of  J/3AVS  Lends  itself  to  incorporation  o:  changing 
requirements  (such  as  J 73*  language  revision)  and  upgrading  capabilities. 
For  example,  anticipated  changes  in  the  JOVIAL  J 73  language  specifi¬ 
cation  (scheduled  to  be  resolved  by  July  1,  1930)  are  expected  to  affect 
only  the  syntax  analyzer.  Upgrades,  such  as  adding  a  conf igurat ion 
management  capability  or  adding  a  target  machine  statement  simulator, 
would  be  performed  by  adding  new  functional  segments.  The  database  is 
designed  so  that  new  tables  of  information  can  be  easily  added,  and  the 
database  manager  does  not  depend  upon  the  type  cf  information  stored  in 
the  tables. 


b  FUTURE  EFFORT 

There  are  five  techniques  for  software  verification  that  should  be 
considered  for  future  implementation  in  J73AVS.  The  two  more  important 
areas  are  test  data  generation  and  instruction-level  simulation.  Test 
data  generation  would  be  a  valuable  assistant  for  all  applications  to 
JOVIAL  J 73.  Instruction-leve l  simulation  for  the  purpose  of  analyzing 
size,  accuracy,  and  timing  for  target  machines  would  be  beneficial  for 
real-time  applications,  such  as  avionics. 
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Additional,  completely-automatable  facilities  are  code  auditing, 
physical  units  consistency  checking,  and  assertion  translation  using  a 
precompiler.  Detection  of  certain  "dangerous"  coding  practices  is 
included  in  the  J73AVS  static  analyzer.  It  cannot  be  too  strongly 
stressed  that  such  practices  should  be  retained  only  for  compatibility 
with  existing  cooe;  :se  in  new  applications  should  be  prohibited  except 
where  extreme  requirements  exist  for  time  and  space  efficiency.  When 
J73  becomes  a  familiar  language,  coding  standards  should  be  specified  by 
the  JOVIAL  User's  Group  (an  Air-Force-sponsored  group  of  interested 
individuals  from  industry,  Government,  and  the  military)  and  included  in 
J73AVS.  Units  consistency  checking  is  already  performed  in  AVS  tools 
such  as  SQLAB.  The  addition  of  this  facility  to  J73AVS  would  be  a 
small  effort. 

It  has  been  the  practice  at  GRC  to  design  and  develop  automated 
software  tools  using  a  top-down,  modular  approach.  Our  basic  approach 
is  to  isolate  major  functional  blocks  into  software  components  that  have 
well-defined  interfaces.  When  new  or  more  efficient  techniques  are 
developed,  they  are  incorporated  into  the  system  as  additional  or 
replacement  components.  Both  test  data  generation  and  instruction-level 
simulation  can  be  incorporated  into  J73AVS  as  additional  functional 
components . 
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6.1  TESY  DATA  GENERATION 


In  order  to  implement  a  tost  ant  a  generation  system,  the  following 
functional  components  are  required: 

1.  Syntax  analyzer — breaks  incoming  source  text  into  tokens  and 
stores  module,  statement,  and  symbol  information  into  tables 
for  subsequent  use. 

2.  Structural  analyzer — generates  a  directed  program  graph  for 
each  module  based  on  its  control  structure;  saves  the 
control  path  information  in  the  branch  table  for  i.iter  use. 

3.  Pseudo-path  eliminator— -this  component  contains  two 
techniques : 

a.  Acting  on  interactive  command  i  rom  the  user,  it 
eliminates  sequences  of  paths  from  the  test  case 
selection  process  wnich  are  logically  impossible  of 
"uninteresting"  during  a  particular  testing  activity. 

b.  Using  backward  symbolic  execution,  automat  ion i Ly 
determines  and  eliminates  logically  impossible  path 
sequences . 

4.  Reaching  sequence  generator — generates  reaching  sequences 
according  to  (1)  interactive  identification  by  the  user  of 
starting  and  stopping  branches  or  (2)  algorithmic  iden¬ 
tification  of  the  starting  and  stopping  branches  based  on 
execution  coverage  performance.  Also  generates  individual 
branch  sequences. 

3.  Reaching  sequence  constraint  genera  tot — builds  an  expression 
resulting  from  the  backtracked  reaching  sequence.  Also 
analyzes  individual  branch  sequences. 
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Constraint  simplifier — uses  arithmetic,  logical,  and  re¬ 
lational  simplification  to  reduce  t  he  path  sequence  con¬ 
straint  to  a  set  of  inequalities.  This  process  should 


utilize  interactive  assistance  from  the  user  in  terms  of 


additional  simplir icat ion  rules. 


inequality  solver — generates  input  data  for  subsequent 
dynamic  execution  according  to  some  automated  or  inter¬ 
actively-supplied  heuristics.  If  the  set  of  inequalities  is 
nonlinear,  interactive  assistance  will  be  required  to 


determine  solutions. 


Instruinentor — (1)  automatically  stores  software,  "probes" 
into  the  source  code  so  that  coverage  information  can  be 
recorded  during  dynamic  execution,  and  (2)  automatically 
translates  user-supplied  assertion  statements  in  the  source 


code  into  executable  statements. 


Execution  analyzer — processes  the  trace  file  recorded  during 
execution  of  the  instrumented  source  code  to  provide  branch 
and  module  execution  coverage  information. 


Table  builder — builds  certain  tables  such  as  symbol  cross 
reference,  module  dependence,  common  symbols,  etc.  which 
will  be  needed  for  documentation  reports  and  backtracking 


through  the  module  hierarchy. 


Report  generator--produces  a  variety  of  user-specified 
(through  the  command  language)  reports  about  the  char¬ 
acteristics  of  the  test  program  as  a  whole  or  with  respect 
to  specified  target  brandies. 


The  functional  components  briefly  described  above  are  included  in 
Fig.  6.1  which  puts  the  manual,  interactive,  and  automated  capabilities 
into  perspective.  Note  that  the  insertion  of  assertions  (described 
briefly  in  Sec.  4  and  in  detail  in  the  Functional  Description)  is  shown 
as  the  first  activity.  The  power  of  assertions  lies  in  their  ability  to 
provide  functional  information  about  the  program  which  both  the  test 
tool  and  user  can  analyze  to  determine  correctness  of  program  behavior 
and  completeness  in  functional  testing. 

6.2  INSTRUCTION-LEVEL  SIMULATION 

With  the  advent  of  MiL-STD-1 750,  the  military  standard  instruction 
set  for  airborne  computers,  it  is  not  unreasonable  to  consider  the 
incorporation  of  target  machine  requirements  into  a  general-purpose, 
host-operational  tool  like  J73AVS.  Robert  Class  at  the  Boeing  Company 
has  stressed  the  value  of  testing  software  on  the  host  computer  (see 
App.  B).  it  is  his  contention  that  most  errors  in  embedded  systems  can 
be  traced  to  faulty  code  in  the  host  computer.  Further,  it  is  only  on 
the  host  system  that  computer  and  peripheral  resources  for  extensive 
testing  are  available. 

Simulation  of  the  1750  instruction  set  can  be  a  functional 
component  of  J73AVS  which  contains  default  instruction  size,  precision, 
and  cycle  times  for  a  typical  target  machine.  The  user  can  «.  hange  the 
defaults  through  commands  to  represent  actual  processing  requirements  of 
his  target.  User-requested  reports  will  provide  simulated  operational 
measurements  for  the  target  to  determine  if  the  software  meets  size, 
accuracy,  and  timing  requirements. 

6.3  CODE  AUDITING 

Code  auditors  for  assessing  the  compliance  of  programs  with 
certain  standards  are  common  software  support  tools.  Although  dis¬ 
ciplined  programming  policies  are  encouraged,  it  is  clear  from  high 
maintenance  costs  that  such  policies  are  not  always  foi lowed.  Computer 
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Sciences  Corporation  and  TRU  have  used  code  auditors  on  botu  FORTRAN  ana 
assembly  languages  and  have  reported  a  formal  cost  reduction  of  $37,UUU 
by  using  a  FORTRAN  code  auditor  on  one  pro:ect  alone.*  As  soon  as 
JOVIAL  J73  lias  matured  to  a  level  where  pro, rammer t  can  specify  coding 
standards,  they  should  be  incorporated  int  >  the  static  analyzer  com¬ 
ponent  of  J73AVS.  The  user  would  have  the  option  Lo  select  the  code 
auditing  feature. 

Typical,  general  coding  standards  include  the  following: 

-  Length  of  program  units 

-  Nesting  level  of  loops 

-  Calling  arguments  are  not  expressions 

-  In-line  comments  precede  labeled  statements,  conditional 
statements,  and  invocations 


fa.4  UNITS  CONSISTENCY 

Requiring  that  each  local  variable  and  each  global  variable  be 
specified  in  terms  of  the  physical  units  it  represents  (if  any)  allows 
comprehensive  checking  of  the  consistency  of  units.  This  type  of 
checking  is  particularly  relevant  to  technical  software  where  many 
physical  properties  are  represented  and  there  are  many  possibilit.es  of 
confusion  over  units.  Units  can  be  checked  on  a  multi-module  basis  if 
each  module  contains  a  description  ol  the  units  for  each  physical 
variable  ;c  refers  to.  The  form  of  the  description  for  JOVIAL  might  be: 

UNITS  (< variable-list-1  >  =*  <uni ts-express ion-1  >, 

<var iabie-iist -2>  =  <uni ts-expression-2 ' , . . . ) 


K.  K.  Fischer,  "Software  Quality  Assuron-'f*  Tools:  Recent  Experience 
and  Future  Requirements,"  Software  Quality  and  Assurance  Workshop,  Son 
Diego,  November  197b. 
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An  inconsistency  in  units  is  indicated  if  unlike  units  are  added, 
subtracted,  or  compared.  The  physical-units  analysis  compares  the  right 
and  left  side  of  assignment  statements,  the  right  and  left  side  of 
relational  operations,  and  actual  and  formal  parameters.  For  con¬ 
venience  in  stating  UNITS  assertions,  all  constants  are  assumed  to  be 
unit  less,  except  for  rcro,  which  will  match  any  units  expression.  A 


variable  is  declared  unitless  by  stating  that  its  units  expression  is 
the  constant  1,  as  in  UNITS  (Pi  =  1). 


This  capability  is  already  available  in  GRC's  SQLAB  AVS  for 
FORTRAN  and  Pascal.  It  is  also  recommended  for  inclusion  into  the  MUST 
(Multipurpose  User-Oriented  Software  Technology)  program  for  HAL/S 
software.*  This  added  static  analysis  could  be  incorporated  economi¬ 
cally  by  converting  the  existing  method  used  in  SQLAB.  Violations  of 
consistency  would  appear  within  the  current  J73AVS  static  analysis 
report  (see  the  Functional  Description). 


6.3  EXECUTABLE  ASSERTIONS  PRECOMPILER 

A  minor  effort  to  develop  a  JOVIAL  J73  precompiler  strictly  for 
the  purpose  cf  translating  logical  assertions  into  executable  JOVIAL  J73 
code  would  have  major  benefits  in  producing  more  reliable  programs  early 
in  their  development  stage.  The  precompiler  would  exist  as  a  JOVIAL  J73 
program  that  merely  scans  source  code  for  ASSERT  statements  and  trans¬ 
lates  them  into  several  executable  statements,  including  the  TRACE 
directive,  to  report  assertion  violations. 


An  assertion  precompiler  would  be  more  efficient  than  translating 
assertions  to  executable  code  by  instrumentation,  since  the  precompiler 
does  not  require  the  syntax  and  structural  analysis  and  the  database 
storage  and  manipulation  needed  by  the  multi-purpose  J73AVS. 

*  R.  N.  Taylor,  Integrated  Testing  and  Verification  System  for  Research 
Flight  Software  -  Design  Document,  Boeing  Computer  Services  Company, 
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JAVS  Technical  Report:  Vol.  1  "User's  Guide"  1975,  1976,  1978 
JAVS  Technical  Report:  Vol.  2  “Reference  Manual*'  1975,  1976,  1978 
JAVS  Technical  Report:  Vol.  3  "Methodology  Report*'  1976 
Methodology  for  Comprehensive  Software  Testing  1975 

JAVS  Computer  Program  Documentation  System  Design  and  Implementation 
Manual  1975,  1976,  1978 

JAVS  Final  Report  1975,  1976,  1978 

General  Research  Corporation 
Santa  Barbara,  California 

The  JAVS  (JOVIAL  Automated  Verification  System)  and  testing 
methodology  were  developed  for  the  Air  Force  as  a  near-term  solution  to 
the  problem  of  testing  JOVIAL  J3  software.  The  requirements  for  the 
tool  were  to  provide  an  automated  mechanism  for  measuring  the  thorough¬ 
ness  of  testing  and  assisting  with  generating  new  test  cases  to  increase 
the  level  of  testedness.  The  resulting  tool  has  the  following  func¬ 
tional  capabilities: 

1.  Recognize  JOVTAL  J3  source  text  with  very  few  language 

restrictions  and  build  a  database  for  up  to  250  invokable 
modules  with  no  limit  on  number  of  statements. 

2.  Using  the  database,  identify  potential  structural  infinite 

loops  and  unreachable  code,  insert  software  probes  at  each 

decision  point,  formulate  software  documentation  reports 
showing  symbol,  statement,  control  path,  module,  and 
inter-module  information. 

3.  When  the  instrumented  modules  are  executed  (with  the 

remainder  of  the  program,  if  the  entire  program  is  not 
instrumented),  provide  statement  and  branch  coverage 
information  and  module  execution  timing  data. 
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4.  Provide  lists  of  branches  not  executed  by  each  test  case  and 
the  sequences  of  branches  required  to  be  executed  in  order 
to  reach  the  unexercised  branches. 

5.  Provide  an  assertion  language  to  assist  code  development  and 
testing  whereby  user-supplied  assertion  statements  can  be 
converted  to  standard  JOVIAL  J3  by  JAVS  and  supply  execution 
time  information. 

JAVS  does  not  provide  data  flow  analysis  capabilities  for  con¬ 
sistency  checking,  interface  analysis,  formal  verification,  or  test  data 
generation.  The  1976  published  methodology  report  provides  guidelines 
for  code  development  and  testing  which  are  keyed  to  the  capabilities 
provided  by  the  JAVS  tool.  The  resources  required  by  JAVS  on  the  HIS 
6180  are  summarized  below: 

JAVS  load  size  =  53K  words 

Data  collection  routines  load  size  =  4K  words 
Random  and  sequential  files 

♦Compile  size  of  instrumented  source  =  152  larger  than 

uninstrumented  compile  size 

♦Compilation  time  of  instrumented  source  =  152  longer  than  for 
uninstrumented  source 

♦Execution  time  of  instrumented  source  =  502  longer  than  execution 
of  uninstrumented  source 

♦Coverage  analysis  time  =  3-6  times  execution  time  of  instrumented 
source 

*  These  resource  requirements  are  rough  estimates  which  vary  according 
to  the  control  structure  of  the  program  and  coverage  analysis  options 
requested. 
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Integrated  Testing  and  Verification  System  for  Research  Flight  Software 

-  Design  Document 

Richard  N.  Taylor 

Boeing  Computer  Services  Company 

Seattle,  UA 

NASA  Report  159008,  February  1979 

This  design  document  describes  a  variety  of  software  support  tools 
to  be  included  in  the  MUST  (Multi-purpose  User-Oriented  Software 
Technology)  system  for  HAL/S  software.  The  tools  included  in  this 
design  operate  from  HALMAT,  an  intermediate  representation  of  HAL/S. 
Thus,  the  tools  do  not  have  to  perform  any  parsing.  The  types  of  tools 
are  static  analyzers,  symbolic  executors,  and  dynamic  analyzers.  There 
is  heavy  emphasis  on  static  and  dynamic  assertion  usage  and  statistics 
gathering. 

The  design  recommendation  is  that  small,  modular  facilities  be 
combined  in  a  variety  of  ways  to  accomplish  program  creation  and 
maintenance.  Such  modular  facilities  are: 

-  local  assertions 

-  regional  assertions 

-  internal  documentation 

-  answers  about  previously  written  code 

-  auditor 

-  units  and  scale  checker 

-  cross-reference  map  generator 

-  data  flow  analysis 

-  execution-time  monitoring 

-  instrumentor  for  run-time  monitoring 


Sample  combinations  of  using  these  techniques  are: 


Isolating  an  error  -  dynamic  analysis  with  extensive 
assertion  usage  on  the  suspect  module. 

Initiar  verification  of  new  code  -  both  data-flow  and 
non-data-f lo*-’  static  analysis. 

Broad-based  verification  with  unlimited  resources  -  static 
analysis,  symbolic  execution,  test  coverage. 

Isolation  of  functional  error  -  symbolic  execution  of 
appropriate  paths,  dynamic  analysis. 

Verification  of  previously  verified  modules  -  multi-purpose 
data-flow  analysis  end  static  checking  of  integration 
requirements,  dynamic  analysis  of  concurrent  process 
characteristics. 


Verification  Techniques  for  Flight  Control  Software 
E.  R.  Rang,  J.  K.  Silverman,  J.  J.  Gutmann 
Systems  and  Recearch  Center,  Honeywell 
December  1978 

This  report  describes  several  manual  and  computer-assisted 
techniques  for  the  verification  of  flight  control  software. 
"Verification"  as  used  in  this  report  means  that  the  resulting  system 
functions  as  intended.  Therefore,  the  techniques  described  cover  the 
description  of  requirements,  specifications,  design,  testing,  and 
assertion  verification. 

Flight  control  software  has  characteristics  that  distinguish  it 
from  other  types  of  software.  Among  these  characteristics  are  syn¬ 
chronization,  distributed  processing,  assembly  code,  structurally  simple 
functions,  and  simple  data  types >  The  verification  techniques  recommen¬ 
ded  in  this  report  reflect  these  characteristics. 

The  techniques  described  and  recommended  are: 

1.  HIPO  (Hierarchy  plus  input-process-output)  charts 

2.  Formal  specification  using  SRI's  SPECIAL 

3.  Petri  nets 

A-  Decision  tables  (as  defined  by  Goodenough  and  Gerhart) 

5.  Symbolic  execution 

The  HIPO  charts  provide  a  manual,  disciplined  method  for  stating 
software  requirements,  defining  a  system  design,  and,  when  used  with 
decision  tables,  generating  test  data.  HIPO  charts  allow  for  describing 
the  system  and  its  individual  functions  and  can  be  used  as  a  basis  for 
design  verification.  Since  the  fabrication  of  HIPO  charts  is  manual  and 
there  are  no  enforced  standards  for  theiv  thoroughness,  their  value  is 
completely  dependent  upon  the  generator  of  the  charts. 
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The  use  of  HIPO  charts  facilitates  drawing  Petri  nets,  construct¬ 
ing  decision  tables,  choosing  test  data,  and  performing  manual  symbolic 
evaluation  of  logical  functions.  Petri  nets  can  be  used  to  represent 
interacting  concurrent  processes,  but  they  can  become  complicated  very 
quickly.  The  primary  asset  of  Petri  nets  is  their  usefulness  in 
developing  a  preliminary  design. 


Decision  tables  consist  of  enumerating  each  decision  (condition) 
in  a  program  (Cl,  C2,...),  followed  by  each  action  (Al,  A2,...)  to  be 
undertaken,  and  then  a  set  of  test  data  (Dl,  D2,...)  which  will  exercise 
each  combination  of  conditions  and  alternatives  (collectively  called 


rules).  Since  HIPO  charts  include  conditions  and  actions  with  the 


"process”  section,  decision  tables  can  be  generated  easily.  Theo¬ 
retically,  this  technique  of  manual  test  data  generation  will  exercise 
all  sequences  of  conditions  in  a  program.  There  are  still  two  major 
problems:  (1)  if  loops  are  involved,  there  may  be  an  infinite  number  of 
condition  sequences,  and  (2)  if  "moderate"  data  values  are  selected, 
errors  can  still  exist  which  might  otherwise  be  found  by  stress  testing. 
As  stated  earlier,  two  of  the  characteristics  of  flight  software  are  few 
loops  and  elementary  data  structures  (frequently  just  boolean  struc¬ 
tures).  For  this  software,  then,  a  tool  which  automated  the  development 
of  HIPO  charts,  translated  them  into  decision  tables,  and  generated  the 
test  data  would  be  very  beneficial. 


SRl's  approach  to  verification,  as  described  in  this  report,  is  to 
formalize  the  software  construction  methodology,  thus  allowing  machine- 
assisted  verification.  In  their  formal  language,  SPECIAL,  a  system  is 
described  before  any  considerations  are  included  about  implementation. 
Modules  are  formulated  as  finite-state  automata:  primitive  data  struc¬ 
tures  are  the  states,  operations  are  the  state  transitions,  and  outputs 
are  computed  from  the  inputs  and  final  states.  The  SPECIAL  system  is 
difficult  to  use,  and  the  authors  of  this  report  were  not  convinced  that 


the  results  were  worth  the  trouble. 
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Symbolic  execution  is  used  along  with  user-supplied  assertions  to 
formally  verify  assembly  code  in  the  PLOVER-80  tool.  The  verification 
technique  described  in  PLOVER-80  is  similar  to,  but  not  quite  as 
extensive  as,  that  in  SQLAB,  a  verification  tool  for  FORTRAN  and  IFTRAN. 
PLOVER-80  accepts  a  set  of  assertion  statements  and  the  Intel  8080 
assmbly  code  as  input,  internally  generates  inductive  assertions  with 
new  variable  names,  and  produces  verification  conditions  using  symbolic 
execution  which  must  be  manually  proved  to  be  correct. 


The  good  feature  of  any  assertion-based  tester  or  verifier  is  that 
it  offers  an  additional  means  of  stating  specifications  in  a  module- 
readable  form.  We  have  found  assertions  to  be  extremely  useful  as 
execution-time  checks  during  software  testing.  The  bad  feature  of 
assertions  is  that  they  too  can  be  erroneous,  and  if  a  proof  of  correct¬ 
ness  relies  solely  on  them,  they  had  better  be  correct. 


Boeing  Support  Software  for  Embedded 
Computer  Systems  -  SCP 

Purpose:  1.  Generate  loadable  code 

2.  Support  V  &  V 

3.  Support  maintenance  and  configuration  control 
Capabilities: 

1.  Automated  configuration  management 

2.  J0V1AL/J3B  compiler  with  multiple  code  generators 

3.  Generalized  macro  assembler  with  multiple  targets 

4.  Generalized  link  editor  supporting  multiple  targets 

5.  Specialized  loaders  supporting  multiple  target  interfaces 

6.  Host  computer  statement  level  simulation 

7.  Multiple  target  computer  instruction  level  simulations 

8.  Software  version  comparison  at  source,  object,  load  levels 

9.  Automatic  cross  reference  and  flow  chart  generation 

Design  Concept: 

1.  Open-ended  processor  structure 

a.  Table-driven  common  control  program 

b.  Single  interface  to  host  computer 

2.  Processors  utilize  common  system  routines 

3.  Processors  interface  through  common  database  format 

a.  Extensible  data  formats 

b.  Database  management  utilities 

4.  Machine-independent  processor  design 

a.  Preprocessors  format  machine-dependent  tables 
b«  Special  processing  routines  may  be  added 
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5.  Implementation  in  HOL 


AED  used 


b.  Machine  dependencies  isolated  and  parameterized 


Documentation  Processors: 


1.  Global  cross  reference 


a.  Global  data  dictionary 

b.  Storage  allocation  map 

c.  Data  block  descriptions 

d.  Procedure  called-by/ calls  list 


2.  Flowchart 


Macro-level  J0V1AL/J3B 


b.  AP  assembly  language 


Functional  requirements  for  SCP  are: 

1.  Modification  of  J3B  cross-compiler  to  save  descriptive  and 
set/used  information  for  data  variables 


2.  Modification  of  the  assembler  to  process  operational 
software  data  and  procedure  coding  conventions  and  to  save 
descriptive  and  set/used  information  for  data  variables 


3.  Integrate  the  saved  descriptive  and  set/used  information 
with  output  of  the  linkage  editor  and  source  code  comments 


to  provide  appropriate  formatted  listings 


A.  Allow  text  editing  of  resultant  formatted  listings 


"Real  Time  Software  Debugging  and  Testing:  Proposed  Solutions,"  Robert 
L.  Glass,  The  Boeing  Company,  D180-25249-1,  -2,  -3,  September  1979. 

This  report  shows  that  most  real-time  software  is  tested  in  the 
target,  not  the  host,  computer  environment  even  though  there  are  no 
software  checkout  tools  in  the  target  environment.  However,  since  more 
than  half  of  the  20  projects  surveyed  in  this  report  used  HOL  and  since 
most  errors  are  in  the  source  code  (not  in  generating  the  target's 
object  code  or  in  the  target's  environment),  the  emphasis  of  the 
proposed  solutions  is  on  the  host  computer  environment.  To  check  out 
the  source  code  in  the  host  environment,  both  the  language  debug 
facilities  and  a  software  environment  simulator  must  be  available  on  the 
host. 

For  the  purpose  of  designing  the  J73AVS,  only  the  debug  and  test 
proposed  solutions  (not  those  for  an  environment  simulator)  are  cri¬ 
tiqued.  This  set  of  recommendations  can  be  summarized  as  follows: 

1.  Timing  analysis  can  identify  critical  areas  which  should  be 
recoded  in  assembly  language. 

2.  Self-checking  code,  using  conditional  compilation,  looks  for 
input  data  acceptability,  data  storage  overflow,  assertions 
and  range  checking  and  provides  for  traces  and  dumps. 

3.  Data  contention  analysis  can  prevent  timing  errors  due  to 
parallel  processing. 

4.  Audit  trails  of  data  and  logic  traces  should  be  recorded. 

5.  Fault  tolerance  mechanisims  provide  fcr  defensive  programs. 


6.  A  cross-reference  listing  should  include  structural  rela¬ 
tionships,  data  types,  and  set-used  information.  Both  local 
and  system-level  cross-reference  lists  are  needed. 

7.  Anomaly  checking  such  as  inaccessible  code,  undefined  var¬ 
iables,  type  mismatching  should  be  performed. 

8.  Structural  testing  should  include  logic  branches,  functions, 
and  combinations  of  logic  branches. 

9.  Data  tracing,  procedure  tracing,  and  formatted  snapshot 
dumping  should  be  performed  such  that  data  is  displayed  by 
name,  is  properly  formatted,  and  is  tied  to  program  structure. 

10.  Unsafe  programming  practices  can  be  recognized,  summarized, 
and  reported. 
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Sneak  Software  Analysis 
Boeing  Aerospace  Co. 

Houston,  Texas 

Sneak  analysis  is  a  set  of  manual  and  computer-aided  techniques 
for  uncovering  and  predicting  unplanned  modes  of  operation.  Given 
software  code,  reference  manuals,  requirements  and  specification,  module 
descriptions,  flow  diagrams,  data  structure  definitions,  etc.,  as  input, 
a  manual  encoding  of  the  input  is  made.  Outputs  from  the  Sneak  Software 
Analysis  routines  include:  nodal  set  number  report,  variable  name 
report,  label  name  report,  and  mnemonic  report.  Certain  questionable 
design  practices  are  flagged  such  as  unnecessary  logic  and  unreferenced 
labels  or  variables.  Then  a  manual  verification  process  is  undergone 
using  the  code,  output  reports,  and  specifications  using  a  network  tree 


representation  of  data  and  logic  flow. 

Of  interest  to  AVS’s  are  the  set 
histories: 

1.  Unused  paths 

2.  Inaccessible  paths 

3.  Improper  initialization 

4.  Lack  of  data  storage  usage 
synchronization 

5.  Bypass  of  desired  paths 

6.  Improper  branch  sequencing 

7.  Potential  undesirable  loops 

8.  Infinite  looping 

9.  Unnecessary  (redundant) 
instructions 

10.  Unreferenced  labels 

11.  Bypassed  variable 
initialization 


of  clues  accumulated  through  case 

Implemented  in  J73AVS? 

Yes  -  dynamic  analysis 

Proposed  for  future  effort  (see 
Sec.  6.1) 

Yes  -  static  analysis 
Yes  -  performance  analysis 

Yes  -  dynamic  analysis 
Yes  -  dynamic  analysis 
Yes  -  performance  analysis 
Yes  -  static  analysis 
Set-set-used  detected 

Yes  -  label  report 
Yes  -  static  analysis 


The  Software  Design  and  Verification  System  (SDVS) 

TRW 

Redondo  Beach,  California 

SDVS  is  an  integrated  set  of  non-realtime  software  to  aid  in  the 
development,  coding,  testing,  and  configuration  management  of  avionics 
software  (primarily  DAIS,  the  AFAL  Digital  Avionics  Information  System). 
Its  capabilities  are:  simulation  of  DAIS  processors,  automated  configu¬ 
ration  management  of  mission  software ,  automatic  control  of  simulation 
runs,  editing  and  processing  of  data  generated  by  the  simulation,  and  a 
JOVlAL-like  command  language. 

The  command  language  provides  statements  for  driving  the  sim¬ 
ulation  such  as  assigning  values  to  variables,  transferring  control, 
collecting  data,  evaluating  logical  expressions,  interpreting  post¬ 
processing  requests,  formatting  output,  etc. 

SDVS  requires  a  J73/1  compatible  JOVIAL  compiler  and  a  database 
management  system.  It  currently  operates  on  a  DEC- 10. 

The  facilities  for  debugging  and  validating  avionics  software  are: 

1.  Snapshot/ rollback  -  during  the  course  of  a  simulation,  results 
are  saved  for  a  subsequent  restart. 

2.  Data  recording  -  statement,  transfer,  register,  instruction 
traces;  module  execution  clock  times;  values  of  selected 
variables  traced;  mod' :  data  requested  by  user  printed. 

3.  Post-simulation  run  processing  -  capabilities  to  sort,  edit, 
analyze,  and  output  simulation  data. 


Test  Coverage  Analyzer 
Boeing  Aerospace  Corp. 

Seattle,  Washington 

The  JOVIAL  J73/1  Test  Coverage  Analyzer  provides  segment  execution 
coverage  analysis  as  an  extension  to  the  J73/1  compiler.  The  extent  of 
instrumentation:  (a)  all  branch  points,  (b)  all  branch  points  and  FOR 
loops,  and  (c)  procedures  only,  is  user-specified  as  a  compiler  control 
card  option*  Post-test  analysis  is  performed  by  support  and  system 
routines,  identified  by  the  user  at  link  time. 


An  example  of  the  Test  Coverage  Analyzer's  output  is: 


Procedure 

Name:  APROC 

Stmt.  No. 

Count 

Stmt.  No. 

Count 

Stmt.  No. 

Count 

1 

3 

3 

10 

7 

10 

10 

0 

12 

10 

21 

3 

Procedure 

Name:  DRIVER 

Stmt.  No. 

Count 

Stmt.  No. 

Count 

Stmt.  No. 

Count 

1 

1 

4 

0 

7 

500 

12 

500 

17 

20 

19 

4S0 

25 

10 

31 

1 

The 

resource  impact 

from  using 

the  Test  Coverage  Analyzer 

is: 

1. 

Instrumented  programs  are 

10-30%  larger  than  uninstruaented 

programs.  For 

procedures 

only,  the  overhead  in  size 

is  0-5%. 

The  execute-time  library  is  1100  words. 

2. 

Execution  time 

is  40-60% 

longer  for 

branch  point  ; 

analysis. 

75-100%  for  branch  point  and  FOR-loop  analysis,  and  10-30%  for 
procedure  analysis. 

3.  There  is  no  significant  size  or  time  imj  *.ct  on  the  compiler. 
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The  only  limitation  of  the  Test  Coverage  Analyzer  is:  no  more  than  1000 
segments  pt-  compilation  unit  may  be  analyzed.  This  limitation  may  be 
easily  increased. 


A  System  for  Incrementally  Designing  and  Verifying  Programs,  Vol«  1 

Mark  S.  Moriconi 

University  of  Texas  at  Austin 

and  USC/ Information  Sciences  Institute 

NT IS  Report  ADA055501 

This  report,  a  doctoral  thesis,  presents  a  description  and 
usage-by-example  interactive  dialogue  of  a  verification  system  which 
differs  from  most  other  systems  in  two  ways: 

1.  It  supports  software  design  and  verification  through 
incremental  stages  with  minimal  reprocessing  of  changed 
modules. 

2.  It  provides  a  very  friendly  user  interface  with  a  respon¬ 
sive,  hierarchical  consnand  language. 


The  system,  called  SID,  is  USP-based  and  runs  on  a  PDP-10 
computer.  Most  of  SID  is  written  in  Reduce;  the  rest  is  written  in 
UC1-LISP. 


The  basic  features  of  SID  are  to  accept  designs  of  modules  in 
terms  of  assertions,  determine  what  the  unresolved  external  references 
are,  and  then  automatically  generate  verification  conditions  (VC’s). 
The  system  generates  VC’s  for  paths  that  are  completely  defined, 
ignoring  those  that  are  not.  Thus,  programs  can  be  a  mixture  of 
specifications  only,  complete  program  text,  or  some  in-between  state  of 
development.  Verification  is  performed  by  an  interactive  theorem 
prover.  Each  VC  is  proved  separately.  When  design  changes  are  made, 
the  system  determines  what  new  VC's  need  to  be  generated  and  proves  only 
the  new  ones. 


The  aspects  of  SID  that  are  interesting  in  the  context  of  the 
J73AVS  development  are  the  system's  determination  of  what  has  been 
changed  in  the  software  being  analyzed  and  the  conversational  command 


language.  The  SID  commands  are:  Add,  Delete,  Edit,  Explain,  Help, 
Print,  Prove,  Restore,  Save,  Suggest,  Translate,  VCS,  ?E,?,??.  Host  of 
the  commands  have  subsequent  levels  of  detail,  prompting  the  user  for 
more  information  as  it  is  needed.  As  the  Suggest  and  Explain  commands 
imply,  SID  is  capable  of  providing  a  certain  amount  of  guidance  for 
directing  system  activities  and  giving  explanatory  consents. 


MISSION 

of 

Rome  Air  Devebpment  Center 

RAVC  plans  and  executes  research,  development,  test  and 
selected  acquisition  programs  In  support  oh  Command,  Control 
Comunications  and  Intelligence  [Ch)  activities ,  Technical 
and  engineering  support  within  areas  oh  technical  competence 
ss  provided  to  ESV  Program  Ohhlces  [POs)  and  other  BSD 
elements .  The  principal  technical  mission  areas  are 
communications,  electroma.gnetic  guidance  and  control,  sur- 
veillance  oh  ground  and  aerospace  objects,  intelligence  data 
collection  and  handling,  inhomation  system  technology, 
ionospheric  propagation,  solid  state  seiences,  microwave 
pnysics  and  electronic  reliability,  maintainability  and 
compatibility. 


